website/content/blog/s3-fuse.md

66 lines
2 KiB
Markdown
Raw Permalink Normal View History

2022-02-05 00:50:07 -05:00
---
2023-02-18 13:12:02 -05:00
date: 2022-02-04 23:39:25-05:00
2022-02-05 00:50:07 -05:00
draft: false
math: false
2023-01-05 14:04:45 -05:00
medium_enabled: true
2023-02-18 13:12:02 -05:00
medium_post_id: a6add7081378
tags:
- Storage
title: Mount Object Storage Locally using S3 Fuse
2022-02-05 00:50:07 -05:00
---
On most cloud providers, object storage is cheaper than paying for the equivalent size in block storage. Using FUSE, we can mount S3 compatible object storage with the command `s3fs`. Do note, that there are a few downsides with mounting object storage as documented on their [README](https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md):
- random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
- metadata operations such as listing directories have poor performance due to network latency
- non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
- no atomic renames of files or directories
- no coordination between multiple clients mounting the same bucket
- no hard links
- inotify detects only local modifications, not external ones by other clients or tools
Lets get started by installing `s3fs`:
```bash
# For Fedora
sudo dnf install s3fs-fuse
# For Ubuntu/Debian
sudo apt install s3fs
```
We'll then need to edit `/etc/passwd-s3fs` with our object storage access and secret keys:
```yaml
AccessKeyHere:SecretKeyHere
```
Then we need to set it so that only root can read `/etc/passwd-s3fs`
```bash
sudo chmod 600 /etc/passwd-s3fs
```
Now we can test to see if we can access our bucket:
```bash
sudo s3fs bucketname \
/mnt/mountpoint \
-o url=https://us-east-1.linodeobjects.com \
-o allow_other
```
If we're successful we should be able to access `/mnt/mountpoint`.
To unmount:
```bash
sudo umount /mnt/mountpoint
```
To mount automatically during reboot, add the following to `/etc/fstab`:
```
bucketname /mnt/mountpoint fuse.s3fs _netdev,allow_other,url=https://us-east-1.linodeobjects.com 0 0
```
2023-02-18 13:12:02 -05:00
After editing `/etc/fstab` you can run `sudo mount -a` in order for it to load and mount any new entries.