Linux http file system

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

A filesystem which allows you to mount HTTP directory listings, with a permanent cache. Now with Airsonic / Subsonic support!

License

fangfufu/httpdirfs

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit

Git stats

Files

Failed to load latest commit information.

README.md

HTTPDirFS — HTTP Directory Filesystem with a permanent cache, and Airsonic / Subsonic server support!

Have you ever wanted to mount those HTTP directory listings as if it was a partition? Look no further, this is your solution. HTTPDirFS stands for Hyper Text Transfer Protocol Directory Filesystem.

The performance of the program is excellent. HTTP connections are reused through curl-multi interface. The FUSE component runs in the multithreaded mode.

There is a permanent cache system which can cache all the file segments you have downloaded, so you don’t need to these segments again if you access them later. This feature is triggered by the —cache flag. This is similar to the —vfs-cache-mode full feature of rclone mount

There is support for Airsonic / Subsonic server. This allows you to mount a remote music collection locally.

If you only want to access a single file, there is also a simplified Single File Mode. This can be especially useful if the web server does not present a HTTP directory listing.

Please note if you install HTTDirFS from a repository, it can be outdated.

HTTPDirFS is available as a package in Debian 11 «Bullseye», If you are on Debian Bullseye, you can simply run the following command as root :

For more information on the status of HTTDirFS in Debian, please refer to Debian package tracker

HTTPDirFS is available in the Arch User Repository.

HTTPDirFS is available in the FreeBSD Ports Collection.

Under Ubuntu 18.04.4 LTS, you need the following packages:

libgumbo-dev libfuse-dev libssl-dev libcurl4-openssl-dev uuid-dev 

Debian 11 «Bullseye» and Debian 10 «Buster»

Under Debian 10 «Buster» and newer versions, you need the following packages:

libgumbo-dev libfuse-dev libssl-dev libcurl4-openssl-dev uuid-dev 

The following dependencies are required from either pkg or ports:

gmake fusefs-libs gumbo e2fsprogs-libuuid curl expat 

devel/gmake sysutils/fusefs-libs devel/gumbo misc/e2fsprogs-libuuid ftp/curl textproc/expat2 

Note: If you want brotli compression support, you will need to install curl from ports and enable the option.

You can then build + install with:

Alternatively, you may use the FreeBSD ports(7) infrastructure to build HTTPDirFS from source with the modifications you need.

You need to install macFUSE, cURL, gumbo, and OpenSSL from Homebrew:

brew install macfuse curl gumbo-parser openssl pkg-config 

Apple’s command-line build tools are usually installed as part of setting up Homebrew. HTTPDirFS will be installed in /usr/local .

./httpdirfs -f --cache $URL $MOUNT_POINT 

An example URL would be Debian CD Image Server. The -f flag keeps the program in the foreground, which is useful for monitoring which URL the filesystem is visiting.

-u --username HTTP authentication username -p --password HTTP authentication password -P --proxy Proxy for libcurl, for more details refer to https://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html --proxy-username Username for the proxy --proxy-password Password for the proxy --cache Enable cache (default: off) --cache-location Set a custom cache location (default: "$/httpdirfs") --dl-seg-size Set cache download segment size, in MB (default: 8) Note: this setting is ignored if previously cached data is found for the requested file. --max-seg-count Set maximum number of download segments a file can have. (default: 128*1024) With the default setting, the maximum memory usage per file is 128KB. This allows caching files up to 1TB in size using the default segment size. --max-conns Set maximum number of network connections that libcurl is allowed to make. (default: 10) --retry-wait Set delay in seconds before retrying an HTTP request after encountering an error. (default: 5) --user-agent Set user agent string (default: "HTTPDirFS") --no-range-check Disable the built-in check for the server's support for HTTP range requests --insecure-tls Disable licurl TLS certificate verification by setting CURLOPT_SSL_VERIFYHOST to 0 --single-file-mode Single file mode - rather than mounting a whole directory, present a single file inside a virtual directory. 

For mounting a Airsonic / Subsonic server:

 --sonic-username The username for your Airsonic / Subsonic server --sonic-password The password for your Airsonic / Subsonic server --sonic-id3 Enable ID3 mode - this present the server content in Artist/Album/Song layout --sonic-insecure Authenticate against your Airsonic / Subsonic server using the insecure username / hex encoded password scheme 

-d -o debug enable debug output (implies -f) -f foreground operation -s disable multi-threaded operation 

Airsonic / Subsonic server support

The Airsonic / Subsonic server support is dedicated the my Debian package maintainer Jerome Charaoui.You can mount the music collection on your Airsonic / Subsonic server (*sonic), and browse them using your favourite file browser.

You simply have to supply both —sonic-username and —sonic-password to trigger the *sonic server mode. For example:

./httpdirfs -f --cache --sonic-username $USERNAME --sonic-password $PASSWORD $URL $MOUNT_POINT 

You definitely want to enable the cache for this one, otherwise it is painfully slow.

There are two ways of mounting your *sonic server

In the index mode, the filesystem is presented based on the listing on the Index link in your *sonic’s home page.

In ID3 mode, the filesystem is presented using the following hierarchy: 0. Root

1. Alphabetical indices of the artists’ names
2. The arists’ names
3. All of the albums by a single artist
4. All the songs in an album.

By default, *sonic server is mounted in the index mode. If you want to mount in ID3 mode, please use the —sonic-id3 flag.

Please note that the cache feature is unaffected by how you mount your *sonic server. If you mounted your server in index mode, the cache is still valid in ID3 mode, and vice versa.

HTTPDirFS is also known to work with the following applications, which implement some or all of Subsonic API:

• Funkwhale (requires —sonic-id3 and —no-range-check , more information in issue #45)
• LMS (requires —sonic-insecure and —no-range-check , more information in issue #46. To mount the demo instance, you might also need —insecure-tls )
• Navidrome, more information in issue #51.

If you just want to access a single file, you can specify —single-file-mode . This effectively creates a virtual directory that contains one single file. This operating mode is similar to the unmaintained httpfs.

./httpdirfs -f --cache --single-file-mode https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.0.0-amd64-netinst.iso mnt 

This can be useful if the web server does not present a HTTP directory listing. This feature was implemented due to Github issue #86

You can cache the files you have accessed permanently on your hard drive by using the —cache flag. The file it caches persist across sessions.

By default, the cache files are stored under $/httpdirfs , which by default is $/.cache/httpdirfs . Each HTTP directory gets its own cache folder, they are named using the escaped URL of the HTTP directory.

Once a segment of the file has been downloaded once, it won’t be downloaded again.

Please note that due to the way the permanent cache system is implemented. The maximum download speed is around 15MiB/s, as measured using my localhost as the web server. However after you have accessed a file once, accessing it again will be the same speed as accessing your hard drive.

If you have any patches to make the initial download go faster, please submit a pull request.

The permanent cache system relies on sparse allocation. Please make sure your filesystem supports it. Otherwise your hard drive / SSD will get heavy I/O from cache file creation. For a list of filesystem that supports sparse allocation, please refer to Wikipedia.

Configuration file support

This program has basic support for using a configuration file. By default, the configuration file which the program reads is $/httpdirfs/config , which by default is at $/.config/httpdirfs/config . You will have to create the sub-directory and the configuration file yourself. In the configuration file, please supply one option per line. For example:

--username test --password test -f 

Alternatively, you can specify your own configuration file by using the —config option.

You can control how much log HTTPDirFS outputs by setting the HTTPDIRFS_LOG_LEVEL environmental variable. For details of the different types of log that are supported, please refer to log.h and log.c.

For the normal HTTP directories, this program downloads the HTML web pages/files using libcurl, then parses the listing pages using Gumbo, and presents them using libfuse.

For *sonic servers, rather than using the Gumbo parser, this program parse *sonic servers’ XML responses using expat.

The cache system stores the metadata and the downloaded file into two separate directories. It uses uint8_t arrays to record which segments of the file had been downloaded.

Note that HTTPDirFS requires the server to support HTTP Range Request, some servers support this features, but does not present «Accept-Ranges: bytes in the header responses. HTTPDirFS by default checks for this header field. You can disable this check by using the —no-range-check flag.

Other projects which incorporate HTTPDirFS

• First of all, I would like to thank Jerome Charaoui for being the Debian Maintainer for this piece of software. Thank you so much for packaging it!
• I would like to thank Cosmin Gorgovan for the technical and moral support. Your wisdom is much appreciated!
• I would like to thank Edenist for providing FreeBSD compatibility patches.
• I would like to thank hiliev for providing macOS compatibility patches.
• I would like to thank -Archivist for not providing FTP or WebDAV access to his server. This piece of software was written in direct response to his appalling behaviour.

About

A filesystem which allows you to mount HTTP directory listings, with a permanent cache. Now with Airsonic / Subsonic support!

Источник

How to Quickly Setup HTTP File Server in Ubuntu 20.04

There are quite a few ways to transfer files over home network. One of the easiest ways is to run a single python command in terminal.

Python contains a script that sets up simple http server. No need to install anything, and even no internet connection required.

1. Simply right-click on the folder that contains the files you want to transfer, then select ‘Open in Terminal‘

2. When terminal opens, run the single command:

UPDATE: You can specify the listening port (e.g., 9900) if you get “Address already in use” error, by running command:

python3 -m http.server 9900

Now any device in the same network can access the http file server by going to (change 8000 if you specified another listening port): http://server_ip:8000

In addition, you may run python3 -m http.server & instead to start the http server in background, so the terminal window can be closed.

To check your IP address, go to Settings > Wi-Fi (or Network), click on the gear button after your connected network and check the IPv4 Address.

Ji m

I’m a freelance blogger who started using Ubuntu in 2007 and wishes to share my experiences and some useful tips with Ubuntu beginners and lovers. Please comment to remind me outdated tutorial! And, notify me if you find any typo/grammar/language mistakes. English is not my native language. Contact me via [email protected] Buy me a coffee: https://ko-fi.com/ubuntuhandbook1

Источник