GSM Motion Sensor: A 25% Success Rate

When I decided to make a GSM alarm, I set out a number of goals for myself: to secure my storage unit; to learn more about microcontrollers; to learn more about digital hardware; and to learn about embedded GSM devices.  The only deadline I set for myself was to complete it by December 19 so that I could install it in my storage unit when I went home for the holidays.  The only requirement the alarm needed to satisfy was that when someone opened my storage unit I’d get a text message.

A number of setbacks have made this impossible.  I only completed one of those goals, and that was not the important one.  Some of the setbacks were my fault.  Some were not.  The primary cause was my lack of understanding of how digital hardware works, which is especially frustrating because that’s what I set out to learn.  How could I in two months fail to make any real progress on the main objective?  The truth is that I do not possess the basic knowledge required to bootstrap my learning, and I don’t know how to find it.

I set my goal for the alarm high so that if I met setbacks I could scale back and still meet the basic requirement.  At first it would have an infrared sensor.  Then I could scale back to using a simple switch to complete a circuit and reacting via an interrupt.  Then I could scale back to simply connecting the power to the microcontroller when the door was opened and have it send a text on startup.  At first it would call my phone with a prerecorded message.  Then I could scale back to just sending a text.

I didn’t build an alarm system for my storage unit.  I didn’t learn (much) about microcontrollers.  And I didn’t learn (much) about digital hardware.

From the very beginning I had trouble.  I wasted $20 on a device I couldn’t program without a $60 adapter.  I spoke with some friends and old coworkers and decided to get an Armstrap – an easy-to-program low power microcontroller.  I’d base my alarm off of that instead.  This delay cost me weeks since I had to wait until mid-November to get it from the US.

After I received the microcontroller, it took me almost two weeks to figure out why it was behaving erratically.   I couldn’t get it to reliably detect a circuit opening or closing.  And I couldn’t figure out why the serial port was spewing junk data.  Days went by when posts on forums were unanswered.  Days turned into weeks.  Then suddenly it was December 5 and I had to travelling to Wales for a week then search for an apartment for a week.  And now I’m writing this post on a plane back to Houston, sans alarm.  I didn’t (and still don’t) know where to turn for help.  I know the microcontroller must be capable of doing what I want, I just don’t know how to make it do that.

I think I found a circuit design online that will do what I want.  But after this many failures – this many times thinking “this satisfies my needs exactly” and being utterly wrong – I’m not sure I should be optimistic.  It works for Raspberry Pi.  Why won't it work for my microcontroller?

I was able to connect a GSM chip to my laptop and send and receive basic SMS messages, so I consider that to be the one success.  My one success in four – a 25% success rate.

I’ve taught myself a lot of things.  Via experimentation I taught myself different programming languages and system administration.  Via observation and experience I taught myself about home construction and the basics of spoken languages.  Maybe my mistake was thinking that what I wanted to learn was possible to learn via experimentation alone.

I’ll continue trying to build an alarm.  Maybe I can find someone in Houston willing to spend an hour or two getting it set up just right.  Or maybe it’ll need to wait 8 or 12 months until we go back.

For now, to secure my storage unit I bought a much better lock.  I started watching some lock-picking videos on YouTube and discovered the lock on our unit is pathetically insecure.  The new lock has a thicker shackle, and a much more complex key.  Most advanced lock-pickers can’t pick this lock.  In this case, brute strength won out over technology because I can buy a secure lock on Amazon and have it delivered next-day.

International Voice + SMS Forwarder: SMS Forwarding Part 2

Previously: SMS Forwarding Part 1

I've divided up the process of implementing SMS (text message) forwarding into three parts since there's a lot of "gotchas" in this part.  In the first part, I described the general process.  In this part, I'll provide the steps to implement an auto-responder, plus a walk-through video to demonstrate everything that needs to be done and to point out all the gotchas.  The third post will actually forward the SMS to an email.

There are two general steps in this post: setting up the AWS Lambda function and setting up an API Gateway to call that function.  All the instructions are written out below, followed by the video walkthrough.

A lot of this post is based on the Twilio blog post about creating your own IVR (interactive voice resonse) system with API Gateway and Lambda.

Setting up the Lamba function

1. Log into Amazon Web Serivces and go to the Lambda console.
2. Create a new function based on the 'hello-world-python' template.  (Hint: filter the templates by 'python')
3. Name the function and give it a description.  Paste in the following code for the function:

   1 2 3 4 5 6

   import json def lambda_handler(event, context): print(json.dumps(event, separators=(',',':'))) return {'message': "Sorry, but I haven't set up text forwarding yet. " + "Please email xxxxxxxxxxxxx@gmail.com"}

   This will import the JSON library used for debugging (line 1), define an entry point for the Lambda function (line 3), print some debugging information (line 4, useful for the next post), and then return a simple dictionary with a response message asking to email you (lines 5-6).
4. Create a new IAM role for the function, then save the role and the function.

Setting up the API Gateway

1. Go to the API Gateway console.
2. Create a new API. Under that, create a resource named 'SMS' and then within that resource, create a POST method.
3. Set the POST method to use the Lambda function you created above
4. Update the Integration Request with a mapping template to convert content formatted as application/x-www-form-urlencoded.  Use the following mapping template:

   {
     "reqbody":"$input.path('$')"
   }

   This template converts the POST request data to JSON for the Lambda function.
5. Update the Integration Response to use a mapping template, too. Use this template:

   1 2 3 4 5

   #set($inputRoot = $input.path('$')) <?xml version="1.0" encoding="UTF-8"?> <Response> <Message>$inputRoot.message</Message> </Response>

   This mapping template will convert the JSON from the Lambda function to XML for Twilio.  You can read more about the template language on the Mapping Template Reference.
6. Update the Method Response to send back an XML Content-Type header (application/xml) instead of JSON.
7. Test it out in AWS.
8. Deploy the API.  (It doesn't matter what you name the stage.)
9. Test it for real with curl.  Replace the URL here with your URL, plus the path to the resource you created.  If you used 'SMS' in step 2, the path will be '/sms'.

   curl -X POST -H "Content-Type: application/x-www-form-urlencoded; charset=utf-8" -d"From=%2B2345678901&Body=TestSMS" "https://foobar.execute-api.us-east-1.amazonaws.com/your-api-name/sms"

10. Update Twilio to point to the deployed URL (plus the path of the resource, e.g. https://foobar.execute-api.us-east-1.amazonaws.com/your-api-name/sms)

11. Test it out by sending a text message to your Twilio number.

If you have any trouble, the CloudWatch logs from the API Gateway and from the Lambda function will guide you.

Here's a walkthrough to help you understand all the minutia. The API Gateway console has a lot of unintuitive clicks that you probably will waste time finding if you don't watch the quick walkthrough.

Previously: SMS Forwarding Part 1
Next Update: SMS Forwarding Part 3

International Voice + SMS Forwarder: SMS Forwarding Part 1

Previously: International Call Forwarding

Now that I’ve got voice forwarding working, I’ve got to get SMS (text message) forwarding working.  I’ve decided to split this into multiple parts since there’s a lot to do.

The first part, this post, will cover the general idea of what I’m doing with Amazon Web Services (AWS).  The second part will be a narrated video showing each of the steps to implement an SMS auto-responder, along with the code.  The auto-responder will just send a generic reply asking to email me.  The third part will be the code to actually forward SMS messages to an email address. 

Twilio

Twilio is the service I used to set up the voice forwarding.  It allows you to build custom voice, SMS, and MMS applications.  The general idea is that you give Twilio a URL to load when it gets a call, text, or MMS.  Twilio follows the directions it gets when loading that URL.  I need to set up that URL.  The most basic (useful) direction I can give is to reply with another text message, so that’s what I’ll implement first.

In my previous post, I covered International Call Forwarding.  That was easy because Twilio built tools into their service for forwarding calls.  They don't have forwarding for SMS or MMS.  I’ll use Amazon for generating the response text message, and (in the third part) for forwarding the messages to my email.

The Cloud

For those of you not in the tech world, Amazon is more than just shopping. About 10 years ago Amazon decided to start opening up some of their web site infrastructure to the world.  They rent out web infrastructure, such as computers to do processing and serve webpages, amazingly fast internet connections, firewalls to protect against attackers, and a few other things.  Amazon was the first company to do this on a large scale and they're the most successful by far.  When someone talks about being "in the cloud," they are most likely really talking about "using computers rented from Amazon for a small premium in exchange for a fast internet connection, redundancy, labor, scalability, and solving a whole host of other problems."  Amazon’s offerings are collectively called Amazon Web Services or AWS.

I’m going to create an SMS auto-responder using AWS’s Lambda service and API Gateway service.  Then I'll build a forwarder with their Simple Email Service.

Building an Auto-Responder

Lambda lets you execute small bits of code in response to some trigger.  You don’t need to rent a server from Amazon to run that code and pay for all the idle time.  Generating a reply to a text message is simple, so it’s perfect for Lambda. 

API Gateway will give you a URL that someone can request and then pass that request on to a Lambda function.  There are some language differences to work out regarding the language that Twilio wants to use and the language Lambda wants to use.   I’ll cover that in the second part.

Building a Forwarder

Once I've got the autoresponder working, I can set up SMS forwarding to my email using Amazon Lambda and Amazon's Simple Email Service.

Cost

The cost of running the smallest server with Amazon is about $0.013 per hour, or $9.36 per month.  Since I'm using AWS Lambda, I don't need to run any servers with Amazon.  Generating a few hundred SMS replies via Lambda costs well under one cent per month.  (As an aside, AWS has a free tier of service, where if you use very little, it won’t cost you anything at all.  Some of the services I use will be free, some won’t be.  That $9.36 per month would fall into the free tier, but I’d still need to worry about the maintenance on the server).

There are some additional costs from other unrelated AWS services that I use.  Overall, my Amazon bill is less than $1.00 per month.  Twilio charges $1.00 per month per phone number and $0.0075 per SMS sent or received.  I expect the total cost of SMS forwarding to be under $3.00 per month.  Voice forwarding will cost about the same.

Previously: International Call Forwarding

Next Update: SMS Forwarding Part 2

International Voice + SMS forwarder: International Call Forwarding

Previously: New Project: International Voice + SMS Forwarder

Using Zapier's instructions for How to Build a Smarter International Phone with Twilio in Just 15 Minutes, I was able to set up voice forwarding that would be ready the instant my phone number ported in.

The 30 second version of those instructions is:

1. Set up a Twilio account
2. Apply to port in your phone number
3. Modify your Geographic Permissions to allow international calling.
4. Wait for Twilio to approve your request (this took 24 hours for me)
5. Set up voice forwarding with a Twimlet
6. Wait for the port to complete (3 business days)

I'm going to focus on step 5, which is the most complex part.  Using some tools from Twilio, it's actually pretty easy.

When you view your porting in request, you can set up URLs to configure how to handle voice and SMS.  Twilio will load that URL and it will do some action based on the response.  For voice calls, the following response will forward a call:

<Response>
  <Dial action="/forward?Dial=true" timeout="20">
    33651xxxxxx
  </Dial>
</Response>

It's silly to set up a web server just to serve a simple XML document.  Even something simple like putting an XML file in an AWS S3 bucket is a a bit of work.  So Twilio offers something called Twimlets, which are pre-made services that generate XML documents for basic phone tasks like forwarding, simultaneous ringing, or accepting voicemail.  You use their configuration tool to generate a Twimlet URL, and you just paste that URL into your Voice configuration.

The phone number has the country code (33), followed by the number I want to forward to (6 51 xx xx xx).  You can see the output this URL produces:

mbenza:~$ curl http://twimlets.com/forward?PhoneNumber=33651xxxxxx
<?xml version="1.0" encoding="UTF-8"?>
<Response><Dial action="/forward?Dial=true" timeout="20">33651xxxxxx</Dial></Response>

This XML is the same as the XML example above, just without formatting to make it easy to read.  Once your phone number ports over, Twilio will forward calls from the number you ported in to the number in your Twimlet.

SMS forwarding is a different beast.  Twilio doesn't offer a Twimlet for that.  The main difference between voice and SMS forwarding is that there's a different behavior each time  for SMS; the sender and message changes with every SMS.  For voice calls, the "forward" command automatically encapsulates the caller into the process.  For SMS, there's no forwarding command in Twilio, so you've got to build something to do that for you.

Previously: New Project: International Voice + SMS Forwarder
Next Update: SMS Forwarding Part I

GSM Motion Sensor: Everything arrives

Previously: New Project: GSM Motion Sensor

The PIR (Passive Infrared sensor) + MSP430 development board and the SIM900 development board (GSM / cell service) arrived the day before I moved to France.  I quickly shoved them in my luggage and then ignored them for a few days.  Everything should just work, right?

The Passive Infrared Module is tiny.  Maybe 1" by 1.5".

 

And the SIM900 came, too.  Looking more closely at it, there's a lot of stuff on that board.  I started thinking, "It must take a lot to power all that.  Did I buy two different wrong things?"

My plan was to first get my old MSP430 working – one that I bought in 2012 – then worry about the new one, and finally work on integrating the SIM900.  I looked for Code Composer Studio, the MSP430 development suite and I found out it's not supported on OS X.  But wait!  There's a beta for OS X.  I looked into the beta and found out that it doesn't support my first generation MSP430.

Three year old software usually works just fine. Even 20 year old software works fine on emulators.  Why doesn't three year old hardware work fine?  Electrical engineers, I'm looking at you.

Trying to be positive, and remembering that I've invested money in this, I figure I'll move on to the new MSP430 (with the integrated PIR sensor).  I start looking for the pins to connect, as my wife's cousin John had suggested on Facebook.  They don't exist.  You need a special $70 JTAG programming tool from the manufacturer to program the device.  Or a different one for $115.  More frustrating is the fact that there's a base standard for how to implement JTAG, but different manufacturers expand it in different ways, then charge you an exorbitant amount of money for their specific JTAG programmer.  Not only can I not start small on a known introductory system (my old MSP430), but I can't even start on the more challenging device.

The Olimex PIR device didn't even have the jumpers its manual advertised for switching between TI's JTAG implementation (Texas Instruments designed and manufactures the MSP430) and Olimex's JTAG implementation.

The "jumpers" are solder bridges.  I don't have a 220V soldering iron, so there's no chance I'll be fixing those soon.  Plus I'm not sure I want to buy one that's high enough quality to solder these 1.5mm x 2mm pads right next to an extruded plastic component.

I went back to Facebook to gripe / vent / ask for help and luckily my old National Instruments coworker Craig was able to point me the open source hardware project called Armstrap developed by another NI coworker, Charles.

After a couple emails back and forth with Charles, I decided to buy an Armstrap.  I had it sent to my dad's and I'll get it in two weeks when I visit the US.

The Armstrap is built on a different architecture than the MSP430.  It's got an ARM microprocessor (CPU), specifically the Cortex M4.  The ARM architecture is targeted at a bit of a different market segment than the MSP430.

The MSP430 is very low-level.  The processor on the PIR device I bought is only 16MHz, which is about 160 times slower than the modern laptop processor.  And there's only 2KB of RAM to hold your program (that's about 14 tweets).  But for such a weakling processor, it consumes a minuscule amount of power: about 3mA when active and 0.5µA when sleeping. 

For comparison, a standard smart phone battery provides 2800mAh (2800 milliamps for one hour), so the MSP430 in active mode would last 39 days on a smart phone battery running at full power.  In theory it'd last 600+ years in sleep mode on a smart phone battery, but realistically it'd be much shorter than that, but still over a year.

The ARM on the other hand is a bit higher level.  It's processor runs at 180MHz and consumes about 18mA when active and 12µA when sleeping.  It's got more RAM (192KB - 256KB). Thankfully, almost all of the time will be in sleep mode.  In theory, that's only 26 years of sleep mode.  Practically it'll be much shorter than that, but I think it'll still be over a year.  So I think the Armstrap will work for me.

Previously: New Project: GSM Motion Sensor

Next Update: Getting the Armstrap (est. Thanksgiving)

New Project: International Voice + SMS forwarder

When I moved to France, I got a French phone number.  It works just like an American phone number – I can get calls, SMS (text) messages, and MMS (picture, video, and group) messages.  But it's very expensive for Americans to call or text it.  Most carriers charge about $1-$2 a minute and $0.50 per text and won't even support international MMS.  I don't expect people to call me and spend any more than they're used to.

I want people to be able to call and text me at my old number and have it forward to my new number.  There are services out there that do something like that, but each lacks some feature I need or want.

• Google Voice audio quality is atrocious and depends on a high-bandwidth connection.
• Magic Jack requires a device in your home plugged in, and it's expensive.
• Skype doesn't support SMS forwarding.
• ring.to will forward phone calls to US numbers and can receive SMS, but usability seems terrible.
• Many other services charge a base rate of a few bucks a month ($5-$30), plus a high per-minute voice rate ($0.10+)

So, that means it's time to build my own.  My requirements are:

1. Cheap ($5 or less per month, $0.05 per minute or less)
2. International voice forwarding
3. Good audio quality with minimal delay
4. SMS forwarding, either direct or via email
5. MMS forwarding would be nice.
6. The ability to reply via SMS or MMS would be great.

My original plan was to use a service called CallWithUs, which allows you to set up call forwarding and has great international rates.  I'd port my phone number to Google Voice and then have that forward to my CallWithUs number, which would forward to my French number.  Google Voice has good SMS and MMS support (and can forward calls without getting their poor audio quality in the way), so I started working on that.  It seemed convoluted, but it made sense.

I set up CallWithUs and had it forward to my French number.  Then I went to have Google Voice forward to my CallWithUs number and it wouldn't let me forward.  A post on the Google Voice product forums went unanswered, so I gave up on that.  As is always the case with Google, you get what you pay for.  Too bad I already paid for a number on CallWithUs.

I was at the AWS re:Invent conference a few weeks ago and met some guys from Twilio, a company that aims to make voice, SMS, and MMS internet-friendly.  They mostly target much bigger users who have frequently changing voice and messaging needs, but they have low entry-level pricing and developer-friendly documentation.  I signed up, and got started.

The general process to port in my US phone number, set up call forwarding, and set up SMS forwarding.  For now I'll just have it forward SMS messages to my email.  In the future I might add the ability to reply via email to texts, forward MMS messages to my email, and maybe even support replying via email with MMS messages.

I've only started porting in my US phone number, and I've followed the directions for international call forwarding, so we'll see what happens when the port request goes through.

Next up: International Call Forwarding

New Project: GSM Motion Sensor

My first project will be a motion sensor that will alert me thousands of miles away if it's triggered.  There's actually a really good reason why this is the first thing I'm making.

When I was packing up to move to France, I rented a storage unit.  When I was signing the lease, I overheard the manager talking to a new employee, explaining that theft occasionally happens there and all the stuff she had to do to be vigilant.  That made me think: if someone broke into my storage unit then put a new lock on, it would be months before I knew about it.  The video tapes and access records would probably be long gone.  So, I set out to make something that would let me know as soon as someone entered my storage unit.  

The requirements are simple:

• It must run on a battery for up to 12 months
• It must not rely on WiFi
• It must be relatively cheap

I couldn't find any product that combined these three.  There are GSM game cameras ([1], [2]), but they're very expensive and I couldn't find anything about their battery life.  I considered using my Raspberry Pi with a GSM shield, but the power draw on a Raspberry Pi would kill even the one of the largest USB battery packs you can buy in a week (40000 mAh / 230 mAh for an idle Pi = 174 hours = 7.25 days).

I know a passive infrared (PIR) sensor draws very little power – on the order of 5 µA.  And I know the MSP430 from TI has an ultra low power draw in sleep mode.  Plus, you can get a SIM300 or SIM900 GSM chip that supports serial or SPI communication and can sleep using very little power.

So, it seems like I've got a basic plan sketched out: An MSP430 that will be triggered by a PIR sensor with a SIM300 or SIM900.  I was able to find a MSP430 combined with PIR sensor, so that would eliminate all of the analog electronics.  I bought that, then started asking for advice.

I asked on Facebook, and I got some helpful feedback from one of my old coworkers, Eric, and my wife's cousin, John.  Plus a not-so-helpful comment from my mom.

After I ordered everything with next-day shipping so I could bring it to France, I started thinking if I was getting in over my head.  Is this too big of a first hardware project?  Did I just waste a bunch of money on projects I'm not going to finish?



Next up: GSM Motion Sensor: Everything arrives.