Gotchas in AWS DataSync for in-cloud data transfers in 2021

When I was tasked with installing a DataSync agent on EC2 to transfer data from its filesystem to S3, I had some questions.

“Why not skip EC2 and sync directly from the EFS filesystem?” “How does the networking work if I skip the EC2 step?”

The answers to these questions and a few errors were not apparent to me from StackOverflow or the AWS documentation, but through experimentation and a call with AWS Support, I figured out the solutions.

In the end, it turns out that syncing data from an Amazon EFS file share in a private subnet to Amazon…

Reasoning about AWS costs using the AWS Cost Explorer and the AWS CLI

Photo by StellrWeb on Unsplash

🚂 I’ll walk you through a train of thought. Along the way, you’ll see how I discover what aspects of CloudWatch and API Gateway are generating the most charges in the example AWS account, and how I drill deeper using different filters and even a script:

  1. Log into the AWS Management Console
  2. Go to Cost Management
  3. Open Cost Explorer
  4. I like to drill down to the Last 1 Day to see what is currently generating charges in the AWS account. So select 1D and Apply:

A free trial at Lumigo, Epsagon, DataDog, New Relic or any of them may be costing you tens to thousands of dollars per month.

Are the costs of your monitoring solution as hidden as this reptile? Photo by Lance Anderson on Unsplash

You might have a free tier or a free trial from a monitoring service. But who’s paying to get the data into that service? You are.

How to tell if your free trial is costing you an arm and a leg

Pulling data via CloudWatch GetMetricData API calls is expensive, not only because of the price of $10.00/million metrics requested, but because your 3rd party service could be requesting this data at a high frequency. …

Knowing these 7 differences could help you tighten your company’s security posture…and score higher on the “AWS Certified Security — Specialty” exam

Photo by Michael Dziedzic on Unsplash

Amazon Web Services (AWS) Key Management Service (KMS) is a huge part of security in AWS and on the “AWS Certified Security — Specialty” exam. How do you control access to your keys in KMS?

1. A key can have multiple grants, but only one key policy

Grants can be added and removed for a given key, whereas a key’s key policy can only be replaced. You can run CreateGrant (with an API call) to add specific permissions or RevokeGrant to remove those permissions again. If you use PutKeyPolicy, on the other hand, that replaces the entire key policy.

Write a script to clean the HTML of your article and repost it elsewhere

Photo (…of a water filtration system ?) by Ivan Bandura on Unsplash

As of 2017, Medium no longer offers Custom Domains. They have an API and a few SDKs for publishing to Medium. What if you want to repost your entire Medium content on your personal domain in one batch?

One way is to download your entire content as a .zip file. The API doesn’t seem to let you do such a large export.

After downloading, the parsing can be done in many ways. The way covered here has a Node.js …

Save thousands if not millions of dollars now by exhausting all the built-in cost savings tools AWS provides.

Photo by Rick Tap on Unsplash

Chances are, you can reduce your AWS costs by, let’s say, 10 to 50%. For some companies, that could result in millions of dollars of costs savings every year. Because of Amazon’s economies of scale philosophy, they give you lots of tools to help you save money. Here are a few quick tips on where to start.

Trusted Advisor

AWS Trust Advisor has a Cost Optimization view. Ironically, you need to pay more to save more, that is, this view is only available with…

Using the serverless framework reduced deployment complexity by 97.4%.

A dog is all this photo by Patrick Tomasso has. “sls deploy” is all your deployments need.


Serverless here refers to application components with little to no IT overhead so you can focus on your main logic.

SAM is the Serverless Application Model. It’s an AWS abstraction over AWS CloudFormation that makes common serverless architectures easier to define.

The Serverless Framework is an open-source project maintained by Serverless that not only makes it easier to define but also deploy serverless.

A Serverless Guru is someone who works at Serverless Guru, such as yours truly.

Why convert from SAM? Why not start from scratch?

Converting from SAM to the serverless framework only requires a couple of lines of…

You can use your API 12 Million times per year and pay AWS $0

Photo by Sarah Dorweiler on Unsplash

AWS Lambda

🧞 An AWS Lambda function is flexible:

  • As of Nov. 2018, you can write in any language — even the 1959 language Cobol!
  • You can trigger the code to run from a database action, an API query, a metric in CloudWatch, a 3rd party monitoring app, an IoT device…
  • Even though a Lambda’s max timeout is 15 minutes, it can trigger another Lambda. So you can even do DNA sequencing with Lambda.
  • You’re not locked into AWS. A Lambda just has the core logic — no…

Photo by Thought Catalog on Unsplash

No mail-order brides, no human hair, and no money service businesses. These rules are part of Braintree’s Acceptable Use Policy.

That means you can’t build the next Venmo or PayPal using Braintree.

But what if you want to build that next Redfin or Zillow for real-estate, the next Robinhood or Wealthfront for investing, or the next Simple for banking?

Have you sent money through Venmo or PayPal? Have you bought stuff on Amazon? Sending money and buying stuff with money are different in the eyes of regulators, so there are different APIs for these different use cases.

Which API should you use for regulated financial activities?

Braintree or Stripe…

Yann Stoneman

Solutions Architect @Smartronix. 11x AWS, 6x Azure, and 1x in GCP, Docker & ITIL. Opinions expressed in this blog are my own. Find me on YouTube!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store