Shallow Thoughts

Akkana's Musings on Open Source Computing and Technology, Science, and Nature.

Mon, 23 Feb 2015

Tips for developing on a web host that offers only FTP

Generally, when I work on a website, I maintain a local copy of all the files. Ideally, I use version control (git, svn or whatever), but failing that, I use rsync over ssh to keep my files in sync with the web server's files.

But I'm helping with a local nonprofit's website, and the cheap web hosting plan they chose doesn't offer ssh, just ftp.

While I have to question the wisdom of an ISP that insists that its customers use insecure ftp rather than a secure encrypted protocol, that's their problem. My problem is how to keep my files in sync with theirs. And the other folks working on the website aren't developers and are very resistant to the idea of using any version control system, so I have to be careful to check for changed files before modifying anything.

In web searches, I haven't found much written about reasonable workflows on an ftp-only web host. I struggled a lot with scripts calling ncftp or lftp. But then I discovered curftpfs, which makes things much easier.

I put a line in /etc/fstab like this:

curlftpfs#user:password@example.com/ /servername fuse rw,allow_other,noauto,user 0 0

Then all I have to do is type mount /servername and the ftp connection is made automagically. From then on, I can treat it like a (very slow and somewhat limited) filesystem.

For instance, if I want to rsync, I can

rsync -avn --size-only /servername/subdir/ ~/servername/subdir/
for any particular subdirectory I want to check. A few things to know about this:
  1. I have to use --size-only because timestamps aren't reliable. I'm not sure whether this is a problem with the ftp protocol, or whether this particular ISP's server has problems with its dates. I suspect it's a problem inherent in ftp, because if I ls -l, I see things like this:
    -rw-rw---- 1 root root 7651 Feb 23  2015 guide-geo.php
    -rw-rw---- 1 root root 1801 Feb 14 17:16 guide-header.php
    -rw-rw---- 1 root root 8738 Feb 23  2015 guide-table.php
    
    Note that a file modified a week ago shows a modification time, but files modified today show only a day and year, not a time. I'm not sure what to make of this.
  2. Note the -n flag. I don't automatically rsync from the server to my local directory, because if I have any local changes newer than what's on the server they'd be overwritten. So I check the diffs by hand with tkdiff or meld before copying.
  3. It's important to rsync only the specific directories you're working on. You really don't want to see how long it takes to get the full file tree of a web server recursively over ftp.

How do you change and update files? It is possible to edit the files on the curlftpfs filesystem directly. But at least with emacs, it's incredibly slow: emacs likes to check file modification dates whenever you change anything, and that requires an ftp round-trip so it could be ten or twenty seconds before anything you type actually makes it into the file, with even longer delays any time you save.

So instead, I edit my local copy, and when I'm ready to push to the server, I cp filename /servername/path/to/filename.

Of course, I have aliases and shell functions to make all of this easier to type, especially the long pathnames: I can't rely on autocompletion like I usually would, because autocompleting a file or directory name on /servername requires an ftp round-trip to ls the remote directory.

Oh, and version control? I use a local git repository. Just because the other people working on the website don't want version control is no reason I can't have a record of my own changes.

None of this is as satisfactory as a nice git or svn repository and a good ssh connection. But it's a lot better than struggling with ftp clients every time you need to test a file.

Tags: ,
[ 19:46 Feb 23, 2015    More linux | permalink to this entry | comments ]

Thu, 19 Feb 2015

Finding core dump files

Someone on the SVLUG list posted about a shell script he'd written to find core dumps.

It sounded like a simple task -- just locate core | grep -w core, right? I mean, any sensible packager avoids naming files or directories "core" for just that reason, don't they?

But not so: turns out in the modern world, insane numbers of software projects include directories called "core", including projects that are developed primarily on Linux so you'd think they would avoid it ... even the kernel. On my system, locate core | grep -w core | wc -l returned 13641 filenames.

Okay, so clearly that isn't working. I had to agree with the SVLUG poster that using "file" to find out which files were actual core dumps is now the only reliable way to do it. The output looks like this:

$ file core
core: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), too many program headers (375)

The poster was using a shell script, but I was fairly sure it could be done in a single shell pipeline. Let's see: you need to run locate to find any files with 'core" in the name.

Then you pipe it through grep to make sure the filename is actually core: since locate gives you a full pathname, like /lib/modules/3.14-2-686-pae/kernel/drivers/edac/edac_core.ko or /lib/modules/3.14-2-686-pae/kernel/drivers/memstick/core, you want lines where only the final component is core -- so core has a slash before it and an end-of-line (in grep that's denoted by a dollar sign, $) after it. So grep '/core$' should do it.

Then take the output of that locate | grep and run file on it, and pipe the output of that file command through grep to find the lines that include the phrase 'core file'.

That gives you lines like

/home/akkana/geology/NorCal/pinnaclesGIS/core: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), too many program headers (523)

But those lines are long and all you really need are the filenames; so pass it through sed to get rid of anything to the right of "core" followed by a colon.

Here's the final command:

file `locate core | grep '/core$'` | grep 'core file' | sed 's/core:.*//'

On my system that gave me 11 files, and they were all really core dumps. I deleted them all.

Tags: ,
[ 12:54 Feb 19, 2015    More linux | permalink to this entry | comments ]

Sat, 14 Feb 2015

The Sangre de Cristos wish you a Happy Valentine's Day

[Snow hearts on the Sangre de Cristo mountains]

The snow is melting fast in the lovely sunny weather we've been having; but there's still enough snow on the Sangre de Cristos to see the dual snow hearts on the slopes of Thompson Peak above Santa Fe, wishing everyone for miles around a happy Valentine's Day.

Dave and I are celebrating for a different reason: yesterday was our 1-year anniversary of moving to New Mexico. No regrets yet! Even after a tough dirty work session clearing dead sage from the yard.

So Happy Valentine's Day, everyone! Even if you don't put much stock in commercial Hallmark holidays. As I heard someone say yesterday, "Valentine's day is coming up, and you know what that means. That's right: absolutely nothing!"

But never mind what you may think about the holiday -- you just go ahead and have a happy day anyway, y'hear? Look at whatever pretty scenery you have near you; and be sure to enjoy some good chocolate.

Tags: ,
[ 15:01 Feb 14, 2015    More misc | permalink to this entry | comments ]

Mon, 09 Feb 2015

Making flashblock work again; and why HTML5 video doesn't work in Firefox

Back in December, I wrote about Problems with Firefox 35's new deprecation of flash, and a partial solution for Debian. That worked to install a newer version of the flash plug-in on my Debian Linux machine; but it didn't fix the problem that the flashblock program no longer works properly on Firefox 35, so that clicking on the flashblock button does nothing at all.

A friend suggested that I try Firefox's built-in flash blocking. Go to Tools->Add-ons and click on Plug-ins if that isn't the default tab. Under Shockwave flash, choose Ask to Activate.

Unfortunately, the result of that is a link to click, which pops up a dialog that requires clicking a button to dismiss it -- a pointless and annoying extra step. And there's no way to enable flash for just the current page; once you've enabled it for a domain (like youtube), any flash from that domain will auto-play for the remainder of the Firefox session. Not what I wanted.

So I looked into whether there was a way to re-enable flashblock. It turns out I'm not the only one to have noticed the problem with it: the FlashBlock reviews page is full of recent entries from people saying it no longer works. Alas, flashblock seems to be orphaned; there's no comment about any of this on the main flashblock page, and the links on that page for discussions or bug reports go to a nonexistent mailing list.

But fortunately there's a comment partway down the reviews page from user "c627627" giving a fix.

Edit your chrome/userContent.css in your Firefox profile. If you're not sure where your profile lives, Mozilla has a poorly written page on it here, Profiles - Where Firefox stores your bookmarks, passwords and other user data, or do a systemwide search for "prefs.js" or "search.json" or "cookies.sqlite" and it will probably lead you to your profile.

Inside yourprofile/chrome/userContent.css (create it if it doesn't already exist), add these lines:

@namespace url(http://www.w3.org/1999/xhtml);
@-moz-document domain("youtube.com"){
#theater-background { display:none !important;}}

Now restart Firefox, and flashblock should work again, at least on YouTube. Hurray!

Wait, flash? What about HTML5 on YouTube?

Yes, I read that too. All the tech press sites were reporting week before last that YouTube was now streaming HTML5 by default.

Alas, not with Firefox. It works with most other browsers, but Firefox's HTML5 video support is too broken. And I guess it's a measure of Firefox's increasing irrelevance that almost none of the reportage two weeks ago even bothered to try it on Firefox before reporting that it worked everywhere.

It turns out that using HTML5 video on YouTube depends on something called Media Source Extensions (MSE). You can check your MSE support by going to YouTube's HTML5 info page. In Firefox 35, it's off by default.

You can enable MSE in Firefox by flipping the media.mediasource preference, but that's not enough; YouTube also wants "MSE & H2.64". Apparently if you care enough, you can set a new preference to enable MSE & H2.64 support on YouTube even though it's not supported by Firefox and is considered too buggy to enable.

If you search the web, you'll find lots of people talking about how HTML5 with MSE is enabled by default for Firefox 32 on youtube. But here we are at Firefox 35 and it requires jumping through hoops. What gives?

Well, it looks like they enabled it briefly, discovered it was too buggy and turned it back off again. I found bug 1129039: Disable MSE for Firefox 36, which seems an odd title considering that it's off in Firefox 35, but there you go.

Here is the dependency tree for the MSE tracking bug, 778617. Its dependency graph is even scarier. After taking a look at that, I switched my media.mediasource preference back off again. With a dependency tree like that, and nothing anywhere summarizing the current state of affairs ... I think I can live with flash. Especially now that I know how to get flashblock working.

Tags: ,
[ 17:08 Feb 09, 2015    More tech/web | permalink to this entry | comments ]

Tue, 03 Feb 2015

Studying Glaciers on our Roof

[Roof glacier as it slides off the roof] A few days ago, I wrote about the snowpack we get on the roof during snowstorms:

It doesn't just sit there until it gets warm enough to melt and run off as water. Instead, the whole mass of snow moves together, gradually, down the metal roof, like a glacier.

When it gets to the edge, it still doesn't fall; it somehow stays intact, curling over and inward, until the mass is too great and it loses cohesion and a clump falls with a Clunk!

The day after I posted that, I had a chance to see what happens as the snow sheet slides off a roof if it doesn't have a long distance to fall. It folds gracefully and gradually, like a sheet.

[Underside of a roof glacier] [Underside of a roof glacier] The underside as they slide off the roof is pretty interesting, too, with varied shapes and patterns in addition to the imprinted pattern of the roof.

But does it really move like a glacier? I decided to set up a camera and film it on the move. I set the Rebel on a tripod with an AC power adaptor, pointed it out the window at a section of roof with a good snow load, plugged in the intervalometer I bought last summer, located the manual to re-learn how to program it, and set it for a 30-second interval. I ran that way for a bit over an hour -- long enough that one section of ice had detached and fallen and a new section was starting to slide down. Then I moved to another window and shot a series of the same section of snow from underneath, with a 40-second interval.

I uploaded the photos to my workstation and verified that they'd captured what I wanted. But when I stitched them into a movie, the way I'd used for my time-lapse clouds last summer, it went way too fast -- the movie was over in just a few seconds and you couldn't see what it was doing. Evidently a 30-second interval is far too slow for the motion of a roof glacier on a day in the mid-thirties.

But surely that's solvable in software? There must be a way to get avconv to make duplicates of each frame, if I don't mind that the movie come out slightly jump. I read through the avconv manual, but it wasn't very clear about this. After a lot of fiddling and googling and help from a more expert friend, I ended up with this:

avconv -r 3 -start_number 8252 -i 'img_%04d.jpg' -vcodec libx264 -r 30 timelapse.mp4

In avconv, -r specifies a frame rate for the next file, input or output, that will be specified. So -r 3 specifies the frame rate for the set of input images, -i 'img_%04d.jpg'; and then the later -r 30 overrides that 3 and sets a new frame rate for the output file, -timelapse.mp4. The start number is because the first file in my sequence is named img_8252.jpg. 30, I'm told, is a reasonable frame rate for movies intended to be watched on typical 60FPS monitors; 3 is a number I adjusted until the glacier in the movie moved at what seemed like a good speed.

The movies came out quite interesting! The main movie, from the top, is the most interesting; the one from the underside is shorter.
Roof Glacier
Roof Glacier from underneath.

I wish I had a time-lapse of that folded sheet I showed above ... but that happened overnight on the night after I made the movies. By the next morning there wasn't enough left to be worth setting up another time-lapse. But maybe one of these years I'll have a chance to catch a sheet-folding roof glacier.

Tags: , , ,
[ 19:46 Feb 03, 2015    More photo | permalink to this entry | comments ]

Sat, 31 Jan 2015

Snow day!

We're having a series of snow days here. On Friday, they closed the lab and all the schools; the ski hill people are rejoicing at getting some real snow at last.

[Snow-fog coming up from the Rio Grande] It's so beautiful out there. Dave and I had been worried about this business of living in snow, being wimpy Californians. But how cool (literally!) is it to wake up, look out your window and see a wintry landscape with snow-fog curling up from the Rio Grande in White Rock Canyon?

The first time we saw it, we wondered how fog can exist when the temperature is below freezing. (Though just barely below -- as I write this the nearest LANL weather station is reporting 30.9°F. But we've seen this in temperatures as low as 12°F.) I tweeted the question, and Mike Alexander found a reference that explains that freezing fog consists of supercooled droplets -- they haven't encountered a surface to freeze upon yet. Another phenomenon, ice fog, consists of floating ice crystals and only occurs below 14°F.

['Glacier' moving down the roof] It's also fun to watch the snow off the roof.

It doesn't just sit there until it gets warm enough to melt and run off as water. Instead, the whole mass of snow moves together, gradually, down the metal roof, like a glacier.

When it gets to the edge, it still doesn't fall; it somehow stays intact, curling over and inward, until the mass is too great and it loses cohesion and a clump falls with a Clunk!

[Mysterious tracks in the snow] When we do go outside, the snow has wonderful collections of tracks to try to identify. This might be a coyote who trotted past our house on the way over to the neighbors.

We see lots of rabbit tracks and a fair amount of raccoon, coyote and deer, but some are hard to identify: a tiny carnivore-type pad that might be a weasel; some straight lines that might be some kind of bird; a tail-dragging swish that could be anything. It's all new to us, and it'll be great fun learning about all these tracks as we live here longer.

Tags: ,
[ 10:17 Jan 31, 2015    More misc | permalink to this entry | comments ]

Sun, 18 Jan 2015

Another stick figure in peril

One of my favorite categories of funny sign: "Stick figures in peril". This one was on one of those automated gates, where you type in a code and it rolls aside, and on the way out it automatically senses your car.

[Moving gate can cause serious injury or death]

Tags: , ,
[ 10:19 Jan 18, 2015    More humor | permalink to this entry | comments ]

Thu, 08 Jan 2015

Accessing image metadata: storing tags inside the image file

A recent Slashdot discussion on image tagging and organization a while back got me thinking about putting image tags inside each image, in its metadata.

Currently, I use my MetaPho image tagger to update a file named Tags in the same directory as the images I'm tagging. Then I have a script called fotogr that searches for combinations of tags in these Tags files.

That works fine. But I have occasionally wondered if I should also be saving tags inside the images themselves, in case I ever want compatibility with other programs. I decided I should at least figure out how that would work, in case I want to add it to MetaPho.

I thought it would be simple -- add some sort of key in the images's EXIF tags. But no -- EXIF has no provision for tags or keywords. But JPEG (and some other formats) supports lots of tags besides EXIF. Was it one of the XMP tags?

Web searching only increased my confusion; it seems that there is no standard for this, but there have been lots of pseudo-standards over the years. It's not clear what tag most programs read, but my impression is that the most common is the "Keywords" IPTC tag.

Okay. So how would I read or change that from a Python program?

Lots of Python libraries can read EXIF tags, including Python's own PIL library -- I even wrote a few years ago about reading EXIF from PIL. But writing it is another story.

Nearly everybody points to pyexiv2, a fairly mature library that even has a well-written pyexiv2 tutorial. Great! The only problem with it is that the pyexiv2 front page has a big red Deprecation warning saying that it's being replaced by GExiv2. With a link that goes to a nonexistent page; and Debian doesn't seem to have a package for GExiv2, nor could I find a tutorial on it anywhere.

Sigh. I have to say that pyexiv2 sounds like a much better bet for now even if it is supposedly deprecated.

Following the tutorial, I was able to whip up a little proof of concept that can look for an IPTC Keywords tag in an existing image, print out its value, add new tags to it and write it back to the file.

import sys
import pyexiv2

if len(sys.argv) < 2:
    print "Usage:", sys.argv[0], "imagename.jpg [tag ...]"
    sys.exit(1)

metadata = pyexiv2.ImageMetadata(sys.argv[1])
metadata.read()

newkeywords = sys.argv[2:]

keyword_tag = 'Iptc.Application2.Keywords'
if keyword_tag in metadata.iptc_keys:
    tag = metadata[keyword_tag]
    oldkeywords = tag.value
    print "Existing keywords:", oldkeywords
    if not newkeywords:
        sys.exit(0)
    for newkey in newkeywords:
        oldkeywords.append(newkey)
    tag.value = oldkeywords
else:
    print "No IPTC keywords set yet"
    if not newkeywords:
        sys.exit(0)
    metadata[keyword_tag] = pyexiv2.IptcTag(keyword_tag, newkeywords)

tag = metadata[keyword_tag]
print "New keywords:", tag.value

metadata.write()

Does that mean I'm immediately adding it to MetaPho? No. To be honest, I'm not sure I care very much, since I don't have any other software that uses that IPTC field and no other MetaPho user has ever asked for it. But it's nice to know that if I ever have a reason to add it, I can.

Tags: , , ,
[ 10:28 Jan 08, 2015    More photo | permalink to this entry | comments ]

Syndicated on:
LinuxChix Live
Ubuntu Women
Women in Free Software
Graphics Planet
DevChix
Ubuntu California
Planet Openbox
Devchix
Planet LCA2009

Friends' Blogs:
Morris "Mojo" Jones
Jane Houston Jones
Dan Heller
Long Live the Village Green
Ups & Downs
DailyBBG

Other Blogs of Interest:
DevChix
Scott Adams
Dave Barry
BoingBoing

Powered by PyBlosxom.