Shallow Thoughts : tags : osmand
Akkana's Musings on Open Source Computing and Technology, Science, and Nature.
Wed, 03 Apr 2024
Several years ago I wrote about
Making
a Land Ownership Overlay with QGIS
and Making
Overlay Maps for OsmAnd.
I've been using that land use overlay for years. But recently
I needed to make several more overlays: land ownership for Utah for a
hiking trip, one for the eclipse, and I wanted to refresh my New Mexico
land ownership overlay since it was several years out of date.
It turns out some things have changed, so here's an update,
starting from the point where your intended overlay is loaded
as a layer in QGIS.
Read more ...
Tags: mapping, GIS, qgis, osmand
[
18:15 Apr 03, 2024
More mapping |
permalink to this entry |
]
Fri, 07 Jan 2022
I record track files in OsmAnd
for most of my hikes. But about a quarter of the
time, when I get back to the car, I forget to turn tracking off.
So I end up with a track file that shows the hike plus at least
part of the car trip afterward. Not very useful for purposes like
calculating mileage.
My PyTopo can do simple
track edits, like splitting the track into two or deleting from some
point to the end. But most of the time it's easier just to edit the
GPX file with a text editor.
Eesh, edit GPX directly?? Sounds like something you oughtn't
do, doesn't it? But actually, GPX (a form of
XML) is human readable
and editable. And this specific case, separating the walking from the
car trip in an OsmAnd track file, is particularly easy, because OsmAnd
helpfully adds a speed to every point it saves.
These instructions seem long, but really, once you've done it once,
you'll realize that it's all very straightforward; explaining the steps
is harder than doing them.
Read more ...
Tags: mapping, osmand, grep
[
14:13 Jan 07, 2022
More mapping |
permalink to this entry |
]
Mon, 15 Apr 2019
Now that I know how to
make
a map overlay for OsmAnd, I wanted a land ownership overlay.
When we're hiking, we often wonder whether we're on Forest Service,
BLM, or NPS land, or private land, or Indian land. It's not easy to tell.
Finding Land Ownership Data
The first trick was finding the data. The New Mexico State Land Office has an
interactive
New Mexico Land Status map, but that's no help when walking
around, and their downloadable GIS files only cover the lands
administered by the state land office, which mostly doesn't include
any areas where we hike. They do have some detailed
PDF maps of
New Mexico Lands if you have a printer capable of printing
enormous pages, which most of us don't.
In theory I could download their 11" x 17" Land Status PDF, convert it
to a raster file, and georeference it as I described in the earlier
article; but since they obviously have the GIS data (used for the
interactive map) I'd much rather download the data and save myself
all that extra work.
Eventually I found New Mexico ownership data at
UNM's RGIS page, which has
an excellent collection of GIS data available for download. Click
on Boundaries, then download Surface Land Ownership.
It's available in a variety of formats; I chose the geojson format
because I find it the most readable and the easiest to parse with
Python, though ESRI shapefiles arguably might have been easier in QGIS.
Colorizing Polygons in QGIS
You can run qgis on a geojson file directly. When it loads it shows the
boundaries, and you can use the Info tool to click on a polygon and
see its metadata -- ownership might be BLM, DOE, FS, I, or whatever.
But they're all the same color, so it's hard to get a sense of land
ownership just clicking around.
To colorize the polygons differently, right-click on the layer name and
choose Properties. For Style, choose Categorized.
For Column, pick the attribute you want to use to choose colors:
for this dataset, it's "own", for ownership.
Color ramp is initially set to random.
Click Classify to generate an initial color ramp, then
click Apply to see what it looks like on the map.
Then you can customize the colors by doubleclicking on specific color
swatches. For instance, by unstated convention most maps show Forest
Service land as green, BLM and Indian land as various shades of brown.
Click Apply as you change colors, until you're happy with the
result.
Exporting to GeoTIFF
Update: In 2024, some of these steps have changed; see
Making OsmAnd Overlays with QGIS (2024 Edition).
You can export the colored layer to GeoTIFF using QGIS' confusing
and poorly documented
Print Composer.
Create one with: Project > New Print Composer,
which will open with a blank white canvas.
Zoom and pan in the QGIS window so the full extent of the image you
want to export is visible. Then, in the Print Composer,
Layout > Add Map. Click and drag in the blank canvas,
going from one corner to the opposite corner, and some portion of
the map should appear.
There doesn't seem to be any way to Print Composer to import your
whole map automatically, or for you to control what portion of the map
from the QGIS window will show up in the Print Composer when you drag.
If you guess wrong and don't get all of your map, hit Delete, switch
to the QGIS window and drag and/or zoom your map a little, then switch
back to Print Composer and try adding it again.
You can also make adjustments by changing the Extents in
the Item Properties tab, and clicking the Set to map canvas
extent button in that tab will enlarge your extents to cover
approximately what's currently showing in the QGIS window.
It's a fiddly process and there's not much control, but when you
decide it's close enough, Composer > Export as Image... and
choose TIFF format. (Print Composer offers both TIFF and TIF; I don't
know if there's a difference. I only tried TIFF with two effs.) That
should write a GeoTIFF format; to verify that, go to a terminal and
run gdalinfo on the saved TIFF file and make sure it says it's
GeoTIFF.
Load into OsmAnd
Finally, load the image into OsmAnd's tiles folder as discussed in the
previous article,
then bring up the Configure map menu and enable the overlay.
I found that the black lines dividing the various pieces of land are
a bit thicker than I'd like. You can't get that super accurate "I'm
standing with one foot in USFS land and the other foot in BLM land"
feeling because of the thick black DMZ dividing them. But that's
probably just as well: I suspect the data doesn't have pinpoint
accuracy either. I'm sure there's a way to reduce the thickness of
the black line or eliminate it entirely, but for now, I'm happy with
what I have.
Update: Here's another, easier, way to show land use on OsmAnd using
overlay tiles from the BLM (in the US):
Adding BLM Land Use Maps to Osmand on Android.
It isn't as general (you can only show something you can get from an
online tiled source) and it updates in real-time, meaning it might use
cellphone data rather than working entirely offline, but it's still a great
option to know about.
Tags: mapping, GIS, qgis, osmand
[
18:13 Apr 15, 2019
More mapping |
permalink to this entry |
]
Wed, 10 Apr 2019
For many years I've wished I could take a raster map image, like a
geology map, an old historical map, or a trail map, and overlay it
onto the map shown in OsmAnd
so I can use it on my phone while walking around.
I've tried many times, but there are so many steps and I never
found a method that worked.
Last week, the ever helpful Bart Eisenberg posted to the OsmAnd list
a video he'd made:
Displaying
web-based maps with MAPC2MAPC: OsmAnd Maps & Navigation.
Bart makes great videos ... but in this case, MAPC2MAPC
turned out to be a Windows program so it's no help to a Linux user.
Darn!
But seeing his steps laid out inspired me to try again, and gave me
some useful terms for web searching. And this time I finally succeeded.
I was also helped by a post to the OsmAnd list by A Thompson,
How to get aerial image into offline use?,
though I needed to change a few of the steps.
(Note: click on any of the screenshots here to see a larger version.)
Georeference the Image Using QGIS
Update in Feb 2024: Several things have changed in QGIS georeferencing
(the version I'm using now is 3.28.15-Firenze),
so note the updated sections below.
The first step is to georeference the image: turn the plain raster image
into a GeoTiff that has references showing where on Earth its corners are.
It turns out there's an open source program that can do that, QGIS.
Although it's poorly documented,
it's fairly easy once you figure out the trick.
I started with the tutorial
Georeferencing Basics,
but it omits one important point, which I finally found in BBRHUFT's
How
to Georeference a map in QGIS. Step 11 is the key: the Coordinate
Reference System (CRS) must be the same in the georeferencing window
as it is in the main QGIS window. That sounds like a no-brainer, but
in practice, the lists of possible CRSes shown in the two windows
don't overlap, so unless you follow BBRHUFT's advice and
type 3857 into the filter box in both windows, you'll likely
end up with CRSes that don't match. It'll look like it's working, but
the resulting GeoTiff will have coordinates nowhere near where they
should be
Instead, follow BBRHUFT's advice and type 3857 into the filter
box in both windows. The "WGS 84 / Pseudo Mercator" CRS will show up
and you can use it in both places. Then the GeoTiff will come out in
the right place.
If you're starting from a PDF, you may need to convert it to a raster format
like PNG or JPG first. GIMP can do that.
So, the full QGIS steps are:
-
Start qgis, and bring up Project > Project Properties.
- Click on CRS, type 3857 in the filter box,
and make sure the CRS is set to WGS 84 / Pseudo Mercator.
- (optional) While you're in Project Properties, you may want to
click on General and set
Display Coordinates Using: Decimal degrees.
By default qgis always uses "meters" (measured from where?)
which seems like a bizarre and un-useful default.
- In Plugins > Manage and Install Plugins...,
install two plugins: Quick Map Services and
Georeferencer GDAL.
Update: both OSM and the Georeferencer are now included by default,
so you no longer have to install these plugins.
- Back in the main QGIS window, go to Web > QuickMapServices
(that's one of the plugins you just installed) and pick
a background map, probably something from OpenStreetMap.
Once you find one that loads, navigate to the area you're
interested in.
Update: Sometimes, including an OpenStreetMap is as easy as clicking on
the OpenStreetMap line in the upper left pane of the QGIS window.
Other times there is no such line, and you have to use
Web > QuickMapServices. I don't know how QGIS decides.
(All the OpenStreetMap tile sets print "API Key Required" all over
everything. You can get an API key from Thunderforest -- I use one
for PyTopo -- but it won't help you here, because nobody seems to
know how to get QGIS to use an API key, though I found dozens of
threads with people asking. If you find a way, let me know.)
-
Start the georeferencer:
Raster > Georeferencer > Georeferencer....
It may prompt you for a CRS; if so, choose WGS 84 / Pseudo Mercator
if it's available, and if it's not, type 3857 in the filter box
to find it.
- In the Georeferencer window, File > Open Raster...
Update: If it isn't showing the files you want to open, you may
need to adjust the
Files of type menu to make sure it includes your file type.
When I ran it just now, I had two JPG files to georeference.
The first time, it showed those files automatically.
When I went to open the second file, it showed nothing about I had
to change Files of type to JPEG JFIF before
I could open the file.
- Select matching points. This part is time consuming but fun.
Zoom in to your raster map in the Georeferencer window and pick a
point that's easy to identify: ideally the intersection of two
roads, or a place where a road crosses a river, or similar
identifying features. Make sure you can identify the same point
in both the Georeferencer window and the main QGIS window
(see the yellow arrows in the diagram).
-
Click on the point in the Georeferencer window, then click "from
map canvas".
- In the main QGIS window, click on the corresponding point.
You can zoom with the mousewheel and pan by middle mouse dragging.
Repeat for a total of four to eight points.
You'll see yellow boxes pop up in both windows showing the matching points.
If you don't, there's something wrong; probably your CRSes don't
match.
-
In the Georeferencer window, call up
Settings > Transformation Settings.
Choose a Transformation type (most people seem to like Thin Plate Spline),
a Resampling method (Lanczos is good), a target SRS (make sure it's
3857 WGS 84 / Pseudo Mercator to match the main window).
Set Output raster to your desired filename, ending in .tiff.
Be sure to check the "Load in QGIS when done" box at the bottom.
Click OK.
- File > Start Georeferencing. It's quite fast, and will
create your .tiff file when it's finished, and should load the TIFF
as a new layer in the main QGIS window.
Check it and make sure it seems to be lined up properly with
geographic features. If you need to you can right-click on the layer
and adjust its transparency.
-
In the Georeferencer window, File > Save GCP Points as...
in case you want to go back and do this again. Now you can close
the Georeferencer window.
- Project > Save to save the QGIS project, so if you want
to try again you don't have to repeat all the steps.
Now you can close qgis. Whew!
Convert the GeoTiff to Map Tiles
The ultimate goal is to convert to OsmAnd's sqlite format, but there's
no way to get there directly. First you have to convert it to map tiles
in a format called mbtiles.
QGIS has a plug-in called QTiles but it didn't work for me: it briefly
displayed a progress bar which then disappeared without creating any files.
Fortunately, you can do the conversion much more easily with
gdal_translate, which at least on Debian is part of
the gdal-bin package.
gdal_translate filename.tiff filename.mbtiles
That will create tiles for a limited range of zoom levels (maybe only
one zoom level). gdalinfo will tell you the zoom levels in the
file. If you want to be able to zoom out and still see your overlay,
you might want to add wider zoom levels, which you can do like this:
gdaladdo -r nearest filename.mbtiles 2 4 8 16
Incidentally, gdal can also create a directory of tiles suitable for
a web slippy map, though you don't need that for OsmAnd. For that,
use gdal2tiles, which on Debian is part of
the python-gdal package:
mkdir tiles
gdal2tiles filename.tiff tiles
Not only does it create tiles, it also includes multiple HTML files
you can use to display those tiles using the Leaflet, OpenLayers or
Google Maps JavaScript libraries. Very cool!
Create the OsmAnd sqlite file
Tarwirdur has written a nice simple Python script to translate from
mbtiles to OsmAnd sqlite:
mbtiles2osmand.py.
Download it then run
mbtiles2osmand.py filename.mbtiles filename.sqlitedb
So easy to use! Most of the other references I saw said to use
Mobile Atlas Creator (MOBAC)
and that looked a lot more complicated.
Incidentally, Bart's video says MAPC2MAPC calls the format
"Locus/Rmaps/Galileo/OSMAND (sqlite)", which might be useful to know
for web search purposes.
Install in OsmAnd
Once you have the .sqlitedb file, copy it to OsmAnd's tiles folder
in whatever way you prefer. For me, that's
adb push file.sqlitedb $androidSD/Android/data/net.osmand.plus/files/tiles
where $androidSD is the /storage/whatever location of
my device's SD card.
Then start OsmAnd and tap on the icon in the upper left for your current mode
(car, bike, walking etc.) to bring up the Configure map menu.
Scroll down to Overlay or Underlay map, enable one of
those two and you should be able to choose your newly installed map.
You can adjust the overlay's transparency with a slider that's visible
at the bottom of the map (the blue slider just above the distance scale),
so you can see your overlay and the main map at the same time.
The overlay disappears if you zoom out too far, and I haven't yet figured
out what controls that; I'm still working on those details.
Sure, this process is a lot of work. But the result is worth it.
Check out the geologic layers we walked through on a portion of
a recent hike in Rendija Canyon (our hike is the purple path).
Tags: GIS, mapping, linux, osmand, qgis
[
19:08 Apr 10, 2019
More mapping |
permalink to this entry |
]
Sun, 14 Aug 2016
I use map tracks quite a bit. On my Android phone, I use
OsmAnd, an excellent open-source
mapping tool that can download map data generated from
free OpenStreetMap,
then display the maps offline, so I can use them in places where
there's no cellphone signal (like nearly any hiking trail).
At my computer, I never found a decent open-source mapping program,
so I wrote my own, PyTopo,
which downloads tiles from OpenStreetMap.
In OsmAnd, I record tracks from all my hikes, upload the GPX files,
and view them in PyTopo. But it's nice to go the other way, too,
and take tracks or waypoints from other people or from the web and
view them in my own mapping programs, or use them to find them when hiking.
Translating between KML, KMZ and GPX
Both OsmAnd and PyTopo can show Garmin track files in the GPX format.
PyTopo can also show KML and KMZ files, Google's more complicated
mapping format, but OsmAnd can't. A lot of track files are distributed
in Google formats, and I find I have to translate them fairly often --
for instance, lists of trails or lists of waypoints on a new hike I plan
to do may be distributed as KML or KMZ.
The command-line gpsbabel
program does a fine job
translating KML to GPX.
But I find its syntax hard to remember, so I wrote a shell alias:
kml2gpx () {
gpsbabel -i kml -f $1 -o gpx -F $1:t:r.gpx
}
so I can just type
kml2gpx file.kml
and it will create
a
file.gpx for me.
More often, people distribute KMZ files, because they're smaller.
They're just gzipped KML files, so use "zip" and "unzip" to
unpack them. In Python you can use the zipfile module.
(Updated to reflect that it's zip, not gzip.)
Of course, if you ever have a need to go from GPX to KML, you can
reverse the gpsbabel arguments appropriately; and if you need KMZ,
run zip afterward.
UTM coordinates
A couple of people I know use a different format, called
UTM,
which stands for Universal Transverse Mercator, for waypoints,
and there are some secret lists of interesting local features passed
around in that format.
It's a strange system. Instead of using latitude and longitude like
most world mapping coordinate systems, UTM breaks the world into 60
longitudinal zones. UTM coordinates don't usually specify their zone
(at least, none of the ones I've been given ever have), so if someone
gives you a UTM coordinate, you need to know what zone you're in
before you can translate it to a latitude and longitude. Then a pair
of UTM coordinates specifies easting, and northing which
tells you where you are inside the zone. Wikipedia has a
map
of UTM zones.
Note that UTM isn't a file format: it's just a way of specifying two
(really three, if you count the zone) coordinates. So if you're given
a list of UTM coordinate pairs, gpsbabel doesn't have a ready-made way
to translate them into a GPX file. Fortunately, it allows a
"universal CSV"
(comma separated values) format, where the first line specifies
which field goes where. So you can define a UTM UniCSV format that looks
like this:
name,utm_z,utm_e,utm_n,comment
Trailhead,13,0395145,3966291,Trailhead on Buckman Rd
Sierra Club TH,13,0396210,3966597,Alternate trailhead in the arroyo
then translate it like this:
gpsbabel -i unicsv -f filename.csv -o gpx -F filename.gpx
I (and all the UTM coordinates I've had to deal with) are in zone 13,
so that's what I used for that example and I hardwired that into my
alias, but if you're near a zone boundary, you'll need to figure out
which zone to use for each coordinate.
I also know someone who tends to send me single UTM coordinate pairs,
because that's what she has her Garmin configured to show her.
For instance, "We'll be using the trailhead at 0395145 3966291".
This happened often enough, and I got tired of looking up the UTM UniCSV
format every time, that I made another shell function just for that.
utm2gpx () {
unicsv=`mktemp /tmp/point-XXXXX.csv`
gpxfile=$unicsv:r.gpx
echo "name,utm_z,utm_e,utm_n,comment" >> $unicsv
printf "Point,13,%s,%s,point" $1 $2 >> $unicsv
gpsbabel -i unicsv -f $unicsv -o gpx -F $gpxfile
echo Created $gpxfile
}
So I can say
utm2gpx 0395145 3966291
, pasting the two
coordinates from her email, and get a nice GPX file that I can push
to my phone.
What if all you have is a printed map, or a scan of an old map from
the pre-digital days? That's part 2, which I'll post in a few days.
Tags: mapping, GIS, osmand
[
10:29 Aug 14, 2016
More mapping |
permalink to this entry |
]
Fri, 16 Aug 2013
Dave and I have been doing some exploratory househunting trips,
and one of the challenges is how to maintain a list of houses and
navigate from location to location. It's basically like geocaching,
navigating from one known location to the next.
Sure, there are smartphone apps to do things like "show houses for
sale near here" against a Google Maps background. But we didn't want
everything, just the few gems we'd picked out ahead of time.
And some of the places we're looking are fairly remote -- you can't
always count on a consistent signal everywhere as you drive around,
let alone a connection fast enough to download map tiles.
Fortunately, I use a wonderful open-source Android program called
OsmAnd.
It's the best, bar none, at offline mapping: download data files
prepared from OpenStreetMap
vector data, and you're good to go, even into remote areas with no
network connectivity. It's saved our butts more than once exploring
remote dirt tracks in the Mojave. And since the maps come from
OpenStreetMap, if you find anything wrong with the map, you can fix it.
So the map part is taken care of. What about that list of houses?
Making waypoint files
On the other hand, one of OsmAnd's many cool features is that it can
show track logs. I can upload a GPX file from my Garmin, or record a
track within OsmAnd, and display the track on OsmAnd's map.
GPX track files can include waypoints. What if I made a GPX file
consisting only of waypoints and descriptions for each house?
My husband was already making text files of potentially interesting houses:
404 E David Dr
Flagstaff, AZ 86001
$355,000
3 Bed 2 Bath
1,673 Sq Ft
0.23 acres
http://blahblah/long_url
2948 W Wilson Dr
Flagstaff, AZ 86001
$285,000
3 Bed 2 Bath
1,908 Sq Ft
8,000 Sq Ft Lot
http://blahblah/long_url
... (and so on)
So I just needed to turn those into GPX.
GPX is a fairly straightforward XML format -- I've parsed GPX files
for pytopo
and for ellie,
and generating them from Python should be easier than parsing.
But first I needed latitude and longitude coordinates.
A quick web search solved that: an excellent page called
Find
latitude and longitude with Google Maps.
You paste the address in and it shows you the location on a map
along with latitude and longitude. Thanks to Bernard Vatant at Mondeca!
For each house, I copied the coordinates directly from the page
and pasted them into the file. (Though that got old after about the fifth
house; I'll write about automating that step in a separate article.)
Then I wrote a script called
waymaker
that parses a file of coordinates and descriptions and makes waypoint files.
Run it like this: waymaker infile.txt outfile.gpx
and it will create (or overwrite) a gpx file consisting of those waypoints.
Getting it into OsmAnd
I plugged my Android device into my computer's USB port, mounted it as
usb-storage and copied all the GPX files into osmand/tracks
(I had to create the tracks subdirectory myself, since I hadn't
recorded any tracks. After restarting OsmAnd, it was able to see all
the waypoint files.
OsmAnd has a couple of other ways of showing points besides track files.
"Favorites" lets you mark a point on the map and save it to various
Favorites categories. But although there's a file named favorites.gpx,
changes you make to it never show up in the program. Apparently they're
cached somewhere else. "POI" (short for Points of Interest) can be
uploaded, but only as a .obf OsmAnd file or a .sqlitedb database, and
there isn't much documentation on how to create either one.
GPX tracks seemed like the easiest solution, and I've been happy
with them so far.
Update: I asked on the osmand mailing list; it turns out that on the
Favorites screen (Define View, then Favorites) there's a Refresh
button that makes osmand re-read favorites.gpx. Works great.
It uses pretty much the same format as track files -- I took
<wpt></wpt> sequences I'd generated with waymaker and
added them to my existing favorites.gpx file, adding appropriate
categories. It's nice to have two different ways to display and
categorize waypoints within the app.
Using waypoints in OsmAnd
How do you view these waypoints once they're loaded?
When you're in OsmAnd's map view, tap the menu button and choose
Define View, then GPX track...
You'll see a list of all your GPX files; choose the one you want.
You'll be taken back to the map view,
at a location and zoom level that shows all your waypoints. Don't
panic if you don't see them immediately; sometimes I needed
to scroll and zoom around a little before OsmAnd noticed there were
waypoints and started drawing them.
Then you can navigate in the usual way. When you get to a waypoint,
tap on it to see the description brieftly -- I was happy to find that
multiple line descriptions work just fine. Or long-press on it to pop up a
persistent description window that will stay up until you dismiss it.
It worked beautifully for our trip, both for houses and for
other things like motels and points of interest along the way.
Tags: mapping, GIS, geocaching, househunting, programming, osmand, pytopo
[
15:58 Aug 16, 2013
More mapping |
permalink to this entry |
]