Key takeaways
- Reducing (or setting) log retention policies is a great way to reduce unnecessary data storage
- Reducing data storage reduces carbon emissions
- Reducing data storage saves money!
Some context
As Agile practitioners, we aim to minimize waste. How does this align with the global sustainability goals? (Read on)
This entry has bubbled up from the Digital Cleanup Day as part of the Agile Alliance Sustainability Initiative.
It’s easy to make the assumption that if our data is “in the cloud,” then it’s better for the environment. Fewer trees are being cut down to print documents, photos, etc. However, we don’t often think of the costs associated with storing our data. Storing data “in the cloud” doesn’t mean it’s sitting in some mythical place that doesn’t consume energy. There are physical machines and storage facilities that need power to run (depending on the size of the facility, this could be a significant amount of power) and a way to regulate the temperature so that the machines and the place they’re stored in don’t overheat.
Think about your own digital storage. Do you store photos and videos in a cloud-based service? How about emails, digital documents, etc. And how often do you take the time to delete those items that are no longer serving you or those accidental photos you took when your finger was in front of the camera lens? Now consider the amount of storage everyone in your family and circle of friends is using to store similar items. How about everyone in your town or city? Very quickly, we can see how enormous the quantity of digitally stored items might be. The cost to keep them stored is not insignificant.
A little story…
Allison Lazarz, Software Test Engineer at Dealer.com
The realization that I consider myself to be sustainably savvy in the physical world, yet don’t put a lot of thought into being sustainably savvy when it comes to my digital storage – along with the fact that I work for a company that stores more terabytes of data than I am privy to – fueled a desire to begin a digital-waste-reduction journey!
At the time of this realization, my company was just about to enter into a “HIP Sprint.” We practice Agile Software Development and the HIP (Hardening, Innovation, Planning) sprint is an opportunity for us to take 2 weeks (the length of time of one “sprint,”) to devote to making our software and processes better. I decided to use this time to see how much of our data storage could be reduced.
Our company stores a lot of data. I was operating on a very small scale, just looking at data storage for the group of teams I work with. I wrote a script to see which of our cloud-based logs (stored in AWS Cloudwatch) were the largest, and which of them didn’t have a retention policy (expiration date/TTL) set on them.
I discovered that we had:
-
- Over 1,900 logs with no TTL, and a few with high TTLs that were multiple terabytes in size.
-
- I tackled just those that had 1+ GB of data stored in them.
-
- By reducing TTLs on large log files, or adding TTLs to log files that didn’t have, and I was able to:
-
- Reduce overall data storage by 540.5 GB/month
- This equates to saving just over 1 ton of CO2 emissions annually and a savings of about 162 kWh/year!
The math on these stats should be taken with a grain of salt, as there are many discrepancies about how many kWh it takes to store a GB or MB of data. It’s similar to the CO2 emissions data. But regardless, we are saving energy as a result of this project! Not only that, but we are saving money because we have less data storage. It’s a win-win!
I can’t help but think about the small dent that my project made, and how much more there is to do! If other teams in my company took a similar approach. If all the companies in my city/state/country took the same approach, etc…
I’m hopeful that my small-scale project can inspire others to do something similar. Small, individual actions can add up to big, large-scale change!
Action time!
Rollup your sleeves, we got you covered from what to Assess, a list of items to Clear up and also to Monitor. Time is NOW.
Assess
- What are your digital assets? List them, including:
- devices (computers, smartphones, tablets),
- online accounts (social media, email, financial), and
- digital files (documents, photos, videos).
- Check for outdated software, unused accounts, duplicate files, and any other digital clutter that needs addressing.
- “Live in the moment!” – the suggestion here is to consider taking fewer photos/videos and, instead, be present in your experiences
- Implement a cloud strategy to “follow the sun”, and use the optimum locations from your cloud service provider where energy use is lower AND renewables are plentiful (you can tap into sites such as Energy Dashboard to understand grid energy source mixture)
Clean up
- Which devices you don’t use? Can you give them a second life?
- Check your computer’s power settings. On Windows, this can typically be found in the Control Panel or Settings menu under “Power & Sleep Settings.” On Mac, navigate to System Preferences > Energy Saver.
- Look for unnecessary files, such as old documents, duplicate photos, and unused applications.
- Do you have digital photos automatically backed up? This can result in loads of images being stored that aren’t important to you! Accidental photos – photos you take to remind yourself of something – like the price tag of an item at a store … things you delete off your phone, perhaps, but then leave sitting around in the cloud. Can you purge some of these files?
- Remember the above also within your emails! For example, this easy-to-use tool released by Google to free up storage.
- Sort out your emails, most old emails you don’t need, and check for big files and the like on them.
- Unsubscribe from the newsletter (every email received and physical mail! is an opportunity to request deletion of your data within their infrastructure within the Right to Be Forgotten, GDPR)
- How about logins you no longer need? (review privacy settings and data deletion policies before)
- What’s on your desktop? delete delete delete?
- Which applications do you use every day, can you check their records and storage? what could be deleted?
- Do you (or does your company) store log data? How long is it being stored? Can that period (TTL/retention policy) be shortened? Could you be logging less data?
- Do you use backups? what is the retention period, if any applied; if not, what actions can be taken to put in place and delete legacy and old stuff
- How many tabs do you have open? How can you clear that up too? (Closing unnecessary applications and browser tabs reduces CPU and memory usage)
- Configure servers to autoscale, and only use the number of servers and other devices needed at any given time
- Remove unused features from products
- Analyze your automated tests to see if any can be deleted, run less frequently, or run on a local server instead of a cloud where that is more efficient
- Talk to other people about your new digital diet habits!
Monitor
- Make it a habit to block 30 minutes a month for a clear-up
- Often from system preferences, you can monitor CPU usage
- Moving forward avoid using images and if you do use low-res
- Do you need to store data indefinitely? If not, can you set a retention policy so old data gets deleted after X days?
- Are you storing logged data? If so, could you reduce the amount of log messages that are being stored? Can you set or shorten the TTL of the log files so that they aren’t being unnecessarily stored?
- Schedule time regularly (weekly, monthly, etc…) to go through stored data and delete items that are no longer serving you
- Discuss with teams how to be “in your face” by default with monitoring dashboards or other ways to keep digital waste top-of-mind
Wrapup/Closure
As Agilists, we already have our skills, mindset, and way of working. It is just up to us to use the tools to leverage the influence we have in our teams to make better choices when we design and create new products and features. If we can do that, we can cut down on a lot of waste. Less stuff. More Joy.
It’s up to us.
