tagline: From openSUSE
This document describes the various ways to mirror the content of download.opensuse.org and how an official mirror can get established.
- 1 What does it need to become a public mirror?
- 2 Rsync servers
- 3 Registering Your Mirror
- 4 Staying informed
- 5 How to set up a mirror
- 6 rsync modules
- 7 Planning: Sizes & Update Frequency
What does it need to become a public mirror?
- It needs diskspace in the range of, at least, 40-60 GB. Depending on what is mirrored.
- It needs quite some bandwidth. The actual amount is hard to predict but e.g. in Germany 1 TB per month should be considered the minimum and is easily reached. It is more relaxed if the mirror can afford 2 TB per month. 10mbit is minimum, 100mbit is better. In some locations, the situation can be different and the requirements can vary locally. In general, the more content is being mirrored, the more traffic is attracted, on the other hand we can control the number of redirects quite well. The presence of ISO images is the largest determinator for caused traffic.
- The hardware or OS doesn't really matter.
The current sizes of the rsync modules listed here: Mirror_Infrastructure#rsync modules But note that it is entirely possible to mirror only parts of a module.
Access for the public: rsync.opensuse.org
This rsync server is open to anyone. It offers public access via rsync protocol to the download.opensuse.org content. Access is usually limited to 50 concurrent connections, so you might not always be able to access it. Some of the mirrors listed here might also offer rsync services.
Access for registered mirrors: stage.opensuse.org
Registered mirrors get access to stage.opensuse.org. This server provides the updated content of download.opensuse.org before the official release and has a higher transfer rate than the public servers. You may want to register for access at mailto:email@example.com, if your mirror has at least a 100MBit connection, and if the conditions outlined in the following paragraph are met.
Conditions for access to stage.opensuse.org
A few words about "staged content" up front. Staged content is content that is not meant to be public yet -- but which we still would like to spread to mirrors already, so that at the time of the public release it is already mirrored, and thereby accessible for many people. So how can that be achieved? We set the permissions of the directory to be protected to 'rwxr-x---' (0750). The directory is then served as part of the tree which is hosted on the stage rsync server. When mirrors sync from it, they will replicate those permissions. And when the to release has come, the directory permissions are changed to rwxr-xr-x (0755), and when the mirrors sync the next time, they catch up with it and the directory becomes accessible on their HTTP/FTP servers as well. This process of release by permission change is sometimes called "bit flip release".
There are some caveats with that, which you (as mirror admin) need to observe:
- run rsync with -p (--perms), so that the permissions are reproduced on the target machine.
- if you run a public rsync server: make sure that your rsync daemon runs under a different user id than the script which pulls the content. Otherwise you might be publicly serving the staged content. You can achieve this, for instance, by setting uid = nobody and gid = nogroup in the respective rsync module.
- run your mirror scripts under a user id different from the one which your HTTP/FTP server runs as. An identical user id would make all files readable for the the HTTP/FTP server. The same effect happens if you run the server as root.
- never run your web server (FTP server / rsync server) as root. A somehow recurrent misconfiguration is, if lighttpd is used, that it is run as root, because the configuration which causes it to run as a different user/group has been forgotten.
You should be subscribed to the mirror mailing list (see bottom of this page), so we can keep you up to date with regard to ongoing release activities. We will inform you of the release schedule, and exact timing of public release -- and you can actively support us in fact.
Registering Your Mirror
In order to redirect clients to your mirror, we need the following:
- email address for contact
- HTTP URL on your mirror (e.g. http://yourmirror.example.com/pub/opensuse/)
- is your web server large-file capable? (to handle images larger than 2 GB in size)
- read-only rsync access for our scanner -- for scanning which we perform to keep our download redirector database up-to-date. It is done from 126.96.36.199/22.
- FTP URL, if you run an FTP server. Can serve as fallback protocol for scanning, if rsync is not available. Otherwise, FTP is not used by openSUSE.
- a name and URL of the operator or sponsor of the mirror, for display in the mirror list.
If you provide this data in writing to mailto:firstname.lastname@example.org, we will add your mirror to our mirror database. The mirror database is used by our download server to actively redirect clients to your server. We attempt to distribute requests on a geographical basis per client IP address. The amount of redirects issued also depends on a score which we will determine together with you, in order to match your capacities.
Furthermore, we actively monitor content on mirrors, so that we redirect only to files which actually exists on them. rsync is the most efficient way to do this; scanning through 300.000 files might take only a few minutes with it. The second best method, if rsync is not available, is via FTP, but it is much less efficient (takes considerably longer and places more load on your server). As last resort, we can fall back to HTTP, if neither rsync nor FTP is available. But it crawls. Thus, please do consider adding an rsync module for opensuse content, which allows for much faster scanning of your server.
You may also want to add your mirror to our official mirror list for the released versions or for the development builds, but those lists are not used for the download redirector and might be phased out later. These are wiki pages, simply hit the "Edit" button at the top ;)
The email@example.com mailing list (previously called firstname.lastname@example.org) is low-traffic and used mainly for announcements. It is also a suitable place for discussions around mirroring openSUSE content, should the need arise. To subscribe, please write to mailto:email@example.com and, since it is a closed list, also send a note to mailto:firstname.lastname@example.org and ask to be added.
The general contact address is: mailto:email@example.com
There is an IRC channel named #opensuse-mirrors at freenode.net.
How to set up a mirror
See here for a howto: Mirror_Setup_Howto
The rsync modules on stage.opensuse.org and rsync.opensuse.org are mostly identical. The former has additional content which is yet to be released, and since the latter syncs from it, there is a short sync time lag between them.
Sizes of the rsync modules are triangulated nightly:
An example of a commandline syncing from a module could look like this:
rsync -rlpt rsync.opensuse.org::opensuse-hotstuff-160gb /srv/pub/opensuse/ --delete-after -hi --stats
modules of main interest:
- opensuse-hotstuff-160gb: The most requested files, which fit into 160 GB. This currently includes the install repo and CD/DVD media of the latest product, its updates, and the most popular other repositories. This is the most suitable module for mirrors with limited disk space. The majority of requests goes on exactly these files.
- opensuse-hotstuff-80gb: An even more restricted selection of most popular files, restricted to 80 GB of space. Use this if your mirror has very limited disk space. Still, the majority of requests goes on the files included in this module, so it is highly useful to mirror "only" these files.
- opensuse-updates: This rsync module provides the /update tree, with official updates for released openSUSE distributions, starting with openSUSE 10.3. (To mirror the updates for older releases, check rsync://ftp.suse.com/suse/update/.)
- opensuse-full: This rsync module provides the complete content of download.opensuse.org, except the SL-OSS-factory directory. The reason to exclude this directory is the high frequency of updates inside. To mirror the SL-OSS-factory directory, we recommend using drpmsync to fetch this directory instead, it decreases the traffic to less than 10% compared to rsync.
- opensuse-full-with-factory: The same as the previous one including the SL-OSS-factory directory containing the Factory Distribution. Again, we do not recommend using this module.
- opensuse-source: This rsync module provides the /source tree, which contains source packages of openSUSE 11.1 onwards. Only available on stage.opensuse.org, but without access restrictions.
- opensuse-debug: This rsync module provides the /debug tree, which contains source packages of openSUSE 11.1 onwards, and includes released updates. Only available on stage.opensuse.org, but without access restrictions.
- buildservice-repos: The complete content
- buildservice-repos-main: Everything, but not the home: projects of the users
Updates do happen all the time, whenever a repository from the Build Service got rebuilt and updated. It is also possible to get the updates pushed.
modules for mirroring the drpmsync tree:
- opensuse-drpmsync: distribution/SL-OSS-factory/drpmsync tree only (OSS part)
- opensuse-drpmsync-nonoss: distribution/SL-Factory-non-oss/drpmsync tree only (non-OSS part)
- opensuse-full-with-factory-drpmsync: The same as opensuse-full-with-factory, including the drpmsync tree.
Use only if you intend to set up your own drpmsync server.
If you want to mirror these trees, you need to be aware that the enormous number of files (~200000) will cause a considerable load on our rsync server. Please avoid it if possible.
drpmsync is a sync services for the Factory Distribution. It does reduce the transfer heavily by transmitting only xdelta data, if the local rpm is not older than 1 month.
drpmsync provides access to the SL-OSS-factory directory only. This directory contains always the latest build of the distribution. This means it can also contain an inconsistent or very broken state. It is useful for developers to fetch the latest code or for testers to validate the latest version.
A drpmsync client can get found in the make deltarpm package from http://download.opensuse.org/tools/drpmsync/ and it can be called with this line for example:
drpmsync drpmsync.opensuse.org:8888/Factory /my/directory
Pushing support for Build Service updates
stage.opensuse.org does also host all content from the Build Service. Since the updates do happen all the time, whenever a new package set got built it is also possible to get the content pushed, instead of polling for it. The obviously requires rsync write access for stage.opensuse.org on your server. The advantages of that method are that
- the mirror is almost always up to date,
- no need to run rsync calls via all repositories. The pushing will only update the repositories which have changed. This does reduce the IO load of the mirror a lot.
- the redirector running at http://download.opensuse.org/ is aware that the packages got updated and can immediately redirect to the mirror.
How to become a pushed mirror?
The usual way (but we can also support a different way) is to open a rsync module on your server, where stage.opensuse.org gets write access. A login and password is optionally possible, but not really needed. Please write a mail to mailto:firstname.lastname@example.org afterwards where you tell us:
- the server name where to push
- the rsync module to be used
- optional account and password
- What you want to mirror (everything or only some certain projects)
- What the public download url will be.
- Any kind of a special wish :)
Then, we need rsync read access to scan your mirror for our download redirector. The download redirector database needs to be updated periodically so it reflects the actual files on your mirror. The scanning happens from 188.8.131.52/22.
Planning: Sizes & Update Frequency
|Size||Path of the subtree|| Frequency of update
|25G||update/10.3||frequently (sync it every 4-24 hours)|
|13G||update/11.0||frequently (sync it every 4-24 hours)|
|4.6G||update/11.1|| frequently (sync it every 4-24 hours)
The update trees will continue to grow.
|228G||repositories (including home:/)||very frequently -- push rsync recommended|
|120G||repositories (without home:/)||very frequently -- push rsync recommended|
|<160G||[opensuse-hotstuff-160gb]||varies (sync it every 6 hours)|
|<80G||[opensuse-hotstuff-80gb]||varies (sync it every 6 hours)|
(as of March 14, 2008)