VPS to VHD

Need to back up your VPS that has little hard drive space to a VHD with the intention of hosting it internally?

Guess what – it’s not fun! Transferring data has to be done through the network, unless you have a second drive attached to the VPS that will handle what you need it do. You can back up to an attached Terminal Services drive, but it has some severe overhead and could take a week or more.

The best option I have found is to attach an Amazon S3 storage bucket. I used a piece of software called TNTdrive (https://tntdrive.com/)  that lets you mount your Amazon bucket directly to Windows as a mapped drive. The best part is, you can back up 50GB in about an hour depending on your host network and hardware speed.

The only issues I encountered were drive space – you need to make sure the available drive space is greater than the used space, as TNTdrive has to cache the information before uploading, in multipart, to Amazon S3.

Social Aggregation

Social network aggregation is the process of collecting content from multiple social network services, such as Instagram, Facebook, Twitter, YouTube, etc. into one unified presentation. The task is often performed by a social network aggregator, which pulls together information into a single location, or helps a user consolidate multiple social networking profiles into one profile.

^ That’s the definition from Wikipedia – and boy are there a few out there now. All very expensive. I bet you they have some pretty heavy overhead costs too!

What we’ve done with Afterglow is let you pull in a bunch of your favourite social feeds into a single page. You can run multiple campaigns on each feed – such as a hashtag contest on Twitter, then an image contest on Instagram, and why not throw in your RSS feed from your blog posts and some Facebook engagements for good measure? I just added #hashtags for Instagram today, and tweaked the instant-loading nature of the homepage so it gives you a fast (better) example of real time updates, rather than having to wait 60 seconds for it to refresh. Turns out people don’t have that kind of attention span. Who would have thought? :)

http://afterglow.me

Pulse

I have been slowly working towards adding Pulse to Afterglow – think of it like a heartbeat, every so often it will re-check the server for updated media posts and automagically display them in the open feed. Real time updates you say? Yes. Yes they are. I’m going to have the Pulse controlled from another variable, similar to debug, maxitems, and sort order. The defaults at this point are still going to be 7 items, debug off, and sort order by date (unix timestamped, of course).

The new server is doing great – very happy with Virpus.

Milestones

So, I’ve been watching the registrations on transfermyemail.ca and find it interesting the countries that I’m seeing there. It has started being used in Argentina, the Philippines, Austria,  Nigeria, and the UK, apart from Canada and the USA.  The consistent story I hear with most users is that they’re moving from an old/slow/bad host to something newer and better, and either aren’t being given the support to migrate cPanel that they need, or they aren’t on cPanel and have no easy email migration path. PERFECT! This is exactly why I launched the site! Still free for now. Try it out!

TransferMyEmail

Made some changes to TransferMyEmail yesterday and today – basically I rewrote the parsing routine for folder structure and output of checkboxes. This was needed because originally I had been using “INBOX” as the root folder, which worked to a certain extent, but then had some feedback from Germany, where I found out “POSTEINGANG” could be used too, and also Gmail has a folder structure that is based partly on IMAP and partly on their tags. So you could have a list of folders, then in the middle a [GMAIL] root folder, then some other folders after that.

Long story short, it should handle anything you throw at it to a depth of 3 subfolders, and still iterate through everything properly.

Check out the project here: http://transfermyemail.ca

Afterglow

I have this social aggregation project I’ve been working on called AfterGlow, and it’s not without its challenges, but tonight I did some work on it on a few areas that have been insufficient up until now. Instagram’s API supports pagination, so I added support for that – full support too, so it will continue down your Instagram feed until it reaches your set maximum results or the end of the feed. I also tweaked the parser for Pinterest to allow for date stamps to be read properly, and time sorting works as a result. They have a unique stamp compared to the other feeds so it took a few extra lines of code – I had only recently switched them from scraping over to RSS. Finally, I edited the moderation side of the admin dashboard so that it would be sorted by date too, and fixed the loading screen so it actually gets injected before the server parses and returns all of the feeds.

Check it out here: AfterGlow.me

IMAP Migration PHP Script v2

A few things have changed since the last time I posted the IMAP migration script, and my wishlist of features is almost complete. Instead of a single file that you have to manually specify variables in, you now have fillable form that uses jQuery.post to send variables to another file (called process.php), which handles the actual migration. What this does is solve the memory issue, as you can specify the incremental value (number of emails it will transfer at a time) and it also makes it a little more user-friendly. I added support for unread emails (instead of just marking everything as read), and changed the imap_body command to “peek” instead of doing a full open, which marks the email as read as it’s transferring. This time around I am not going to post the code, but it will be included in the zip file.

On the todo list is directory traversing for people who organize their emails into folders, and some bug fixes.

Unzip this into a folder on your server, then run “email.php” from your browser.

Download here

 

imapmigration

Commits and Gits

Submitted my first change request/commit for Jetpack, which I already have in use on my site. You’d think having Google+ pages as a link would be something that they’d already have on there? Strange! Find it here: https://github.com/Automattic/jetpack/pull/2654

I went a little further down the blogging road with MainWP and converted my plugin/extension over to a Class, and will also be investigating implementation of codecanyon keys into a plugin… and whether or not it would be worth it. Is there better options? Is there a paid route for plugins that works the best? I dunno, but I’ll figure it out.

IMAP Migration PHP Script

So after a bit of digging I was able to complete a working IMAP mailbox-to-mailbox migration script to help move accounts between servers. A lot of time and effort will be saved, as it is run directly on the server rather than linking your mail client to two accounts and transferring it using drag and drop. You can use it to sync your emails using IMAP, however it won’t check if emails already exist before transferring, so you will end up with duplicates.

It uses imap_append (part of php) including the 5th date variable so there is a 1:1 copy of your email between both servers. The only downside to it at this point is script timeouts and running out of memory on the server, as it uses quite a lot. If you limit it to 1000 emails at a time it has no issues.

Here’s the PHP script if you want to try it out:

<?php

$fromMboxServerPath = “{servera.com/notls/imap:143}”;
$fromMboxMailboxPath = “INBOX”;

$toMboxServerPath = “{serverb.com/notls/imap:143}”;
$toMboxMailboxPath = “INBOX”;

$fromMboxMailAddress = “username”;
$fromMboxMailPass = “password”;

$toMboxMailAddress = “username”;
$toMboxMailPass = “password”;

$fromMboxConnStr = $fromMboxServerPath.$fromMboxMailboxPath;
$toMboxConnStr = $toMboxServerPath.$toMboxMailboxPath;

//Increment these (ie: 1001 to 2000) to continue transferring
$fetchStartSeq = 1;
$fetchEndSeq = 1000;

function myLog($str)
{
echo “Log [“.date(‘Y-m-d H:i:s’).”]: $str\n<BR>”;
}

myLog(“Connecting to mailbox”);

function mboxConn($connstr,$addr,$pass)
{
if(!($mbox = @imap_open($connstr, $addr, $pass)))
{
myLog(“Error: “.imap_last_error());
die;
}
else
{
myLog(“Connected to: $addr $connstr”);
return $mbox;
}
}

function mboxCheck($mbox)
{
if(!($mbox_data = imap_check($mbox)))
{
myLog(“Error: “.imap_last_error());
die;
}
else
{
myLog(“Mailbox check “.$mbox_data->Mailbox.” OK”);
myLog($mbox_data->Nmsgs.” messages present”);
return $mbox_data->Nmsgs;
}
}

$fromMbox = mboxConn($fromMboxConnStr, $fromMboxMailAddress, $fromMboxMailPass);
$toMbox = mboxConn($toMboxConnStr, $toMboxMailAddress, $toMboxMailPass);

$fromMboxCount = mboxCheck($fromMbox);
$toMboxCount = mboxCheck($toMbox);

/**
* Loop on mails
*/

$fetchStartUID = imap_uid($fromMbox,$fetchStartSeq);
if ($fromMboxCount < $fetchEndSeq)
{
$fetchEndSeq = $fromMboxCount;
}
$fetchEndUID = imap_uid($fromMbox,$fetchEndSeq);

/**
* Loop on mails
*/

myLog(“Do stuff and backup from UID [$fetchStartUID] to UID [$fetchEndUID]”);

for ($i=$fetchStartSeq;$i<=$fetchEndSeq;$i++)
{
$pfx = “Msg #$i : “;
$h = imap_header($fromMbox, $i);
$fh = imap_fetchheader($fromMbox, $i);
$headerinfo = imap_headerinfo($fromMbox, $i);
$internal_date=date(‘d-M-Y H:i:s O’,$headerinfo->udate); //important!
$fb = imap_body($fromMbox, $i);
$message = $fh.$fb;

$msgUID = imap_uid($fromMbox,$i);

$struct = imap_fetchstructure ($fromMbox, $i);

/**
* We do some logging
*/

myLog($pfx.”Date: [“.$internal_date.”]”);
myLog($pfx.”UID [“.$msgUID.”] SEQ [“.imap_msgno($fromMbox,$msgUID).”] Flags: [“. $h->Unseen . $h->Recent . $h->Deleted . $h->Answered . $h->Draft . $h->Flagged.”]”);
myLog($pfx.”From: [“. htmlspecialchars($h->fromaddress) . “] To: [“.htmlspecialchars($h->toaddress).”]”);
myLog($pfx.”Subject: [$h->subject]”);

/**
* Transfer email using imap_append – All will be marked as read using 4th variable SEEN
*/
if (!($ret = imap_append($toMbox,$toMboxServerPath.$toMboxMailboxPath,$message,”\SEEN”,$internal_date)))
{
myLog(“Error: “.imap_last_error());
die;
}
else
{
myLog(“everything ok, mail [$fetchStartUID:$fetchEndUID] downloaded and moved in $newMailboxNameMOVE”);
}
}

/**
* End
*/

imap_close($fromMbox);
imap_close($toMbox);

myLog(“Connection closed”);

?>