Docker image to set up a development environment easily

From this discussion, we’d like to have a docker image to ease the installation of a development environment, so the first step to contribution is easier for new comers.

@csammy already create a docker image. This image needs to be official so we can recommend it in the guides and everywhere else. What we need to do to achieve this is:

  • Find long term maintainers. I knew @koehn is familiar with docker as he already maintains an unofficial diaspora docker image for his production setup so I proposed him to join and he looks okay
  • Check the current state of the image, I don’t know how clean it is, if it does everything we want. Opinions there please?
  • Move the repository to the github.com/diaspora/ so it is clearly marked as official, give permissions there to CSammy and Koehn, and to the current core team I guess (so at least @denschub, @jhass and @supertux88)
  • Edit the wiki to change the guide pages and the getting started page to indicate that the easiest way is now to use the docker image

Did I miss something?

It would be nice to have that done before the publication of our blog post calling for contributors.

2 Likes

As @csammy wrote below your post on diaspora*, he is willing to continue and finish the work. Let’s give him the time he needs, it doesn’t really matter if we wait a few months more. As for making anything official, I usually like postponing even thinking about that to a point where we actually have something to make official:wink:

The following has to be taken with some grains of salt (and pepper), as I only started understanding and learning Docker recently (a year ago or so) and the amount of Docker images I produced for actual use is still single-digit, so I may be missing something in terms of maintainability etc.

We want to have something for development, so there are a few things I particularly dislike about @koehn’s container. The implementation may be fine for someone to use in production (although I still feel bad about this, but that’s a different discussion), but it is not suitable for a development env:

  • Brads image has the sources inside the container. On build time, you pass in a Git URL, which the script will then clone inside the container. If someone wants to run a custom code base, they either have to commit it to some Git repo (which makes testing a pain) or manually bind the sources from a local directory inside the container (which requires knowledge of Docker). Sammys image always makes a git clone into a local directory, mounts the directory into the container and runs from there. Therefore, editing the source files is easy, and even auto-reloading works.
  • The only reason we have people install RVM in production is so that we can control the Ruby version. For Docker, there are base images for every Ruby version out there, so that is not needed. Using the Ruby base images makes it much easier to update Ruby if needed, and also test with different versions.
  • As for database support, Brads image only supports PG. This, again, is okay for production, but there are valid cases where we need to switch to MySQL for development. Sammys image sets up both, and decides which DB server to start based on the current database.yml. For developers, all they have to do is change their database config and run docker-compose restart.
  • The wrapper script makes it easy to run common commands like running tests or migrating the database inside the container without any hassle.

Ultimately, Brads image is fine for production installations if someone feels like it, but IMO, Sammys approach is much more suitable for providing a development environment. That’s probably because Brad designed the image for production setups, while Sammy designed a development environment, eh… :slight_smile: I’d even go so far to say that we could tell someone with zero Docker experience to “clone this repo, run this script, and do development work” without any issues whatsoever, as the wrapper resolves all the points where someone would actually need to know Docker.

As Sammy started, and clearly states he is interested in finishing the work, I see no reason why we shouldn’t let him do that. If we have to wait a few months before publishing the blog post, so be it. It’s not like we have something to lose.

1 Like

I really have nothing to disagree about with @denschaub’s assessment. I’m happy to help @csammy with his work. The image I made follows the official Debian instructions (hence rvm) and goes out of its way to remove everything not needed for production use, to keep the image size small and to reduce the attack footprint.

1 Like

When you all talk about “Docker image” here, are you talking about this?

@denschub sorry if my post made you think I wanted to start from koehn image, that’s not what I was trying to say. I was saying we use CSammy image, and that koehn could help improve / maintain it, as one of your concern was finding long term maintainers.

@balacobaco yes

Sammys image is almost done, I think the only thing that’s still open is testing how/if it works with spring, or if there are any problems related to spring (or maybe we can just ignore spring, but it would be cool to have faster restarts and tests). But I’m not sure anymore. Otherwise this image is very much optimized for development and the wrapper script has all commands needed for development.

About making it official: I told Sammy to create a PR when he thinks it’s ready, because I want the wrapper-script to be inside the main repo, so you only need to clone the repo (the main repo or your fork) and are ready to go. You don’t need to clone two repos and link them somehow together. But Sammy isn’t ready yet, that’s why there is no PR yet (but as @denschub said, there is no need to rush).

About maintaining it: I don’t think it would need much maintenance when it’s done, because it just works as a local dev env, and I don’t need to change much there either. So I think in the rare event that something needs to be changed, I need to do it to my dev env too, so I probably could fix it in the container at the same time if nobody else can maintain it (I already contributed a bit to the container :slight_smile: ). I think it’s useful when you have development experience to maintain it, so you know what’s needed for a development container. That’s why I gave Sammy a lot of feedback in the beginning, about what’s needed for the wrapper script to be useful as a dev container (because it’s mostly very much the opposite of a production container).

2 Likes

So, I finally got around to it and created a pull request. I’m looking forward to your comments, suggestions and contributions.

Specifically, I’m having trouble with making the Docker-setup-specific YAML config files maintainable in a better way. After struggling with it for >6h I decided to go with the simple approach and hope somebody else will come up with a better idea :slight_smile:

3 Likes

Small “up” to CSammy post above, he feels the PR is currently ready to be merged. If anyone wants to do some final feedback, it’s right now :wink: @koehn maybe?

I’d like to put in a tid-bit from something I experienced a bit earlier. Please have your docker request the user to enter the ports they’re listening on, as some users (like myself) have more than one ruby on rails instance and more than four or five different websites running on their servers that need specific identification. Docker just runs through its settings if the user doesn’t have the chance to configure them before-hand. While understandably one could simply edit the docker config file itself, I’m asking on behalf of the ‘noob-ish’ new sys admins for whom the docker image is created for in the first place :slight_smile:

The docker image is neither created to be run on servers nor is it created for sysadmins. It’s created for developers.

Maybe it’s useful to change the port for development too, but I think it’s only a nice to have, and if it is really needed one can still change the docker-compose file. But maybe we can add a parameter to the script?

1 Like

The Docker image is marked “ONLY FOR DEVELOPMENT” in so many places, I don’t even know how often. I was warned that people may try to use it for production in spite, and did not really believe it. You have made me a believer.

Do me a favour, and forget about the Docker image. It does not exist for anything else than local testing and development.

1 Like

I’m fairly confident that my request should apply for both development and production. It should be noted that developers can also have other platforms for development as well, also utilizing a docker image.

I am fairly confident that this thread is called “Docker image to set up a development environment easily”.

The current implementation linked in this thread is marked for development environments and that’s the way it is for now. There are several concerns about using Docker for production pods, like the fact that nobody has figured out how to run Postgres updates in a reliable manner, nobody did come up with a mechanism to make sure manual update steps are completed before starting up a half-updated instance and thus breaking everything, nobody has figured out how to teach podmins about the security concerns about running Docker in production on a server with direct internet access, and a lot of other concerns that you can find by using the search feature here and on GitHub.

Given the number and the size of those concerns, which get even more weight given a Docker container promoted as production-ready would attract more podmins who are not sure what they are actually doing because they just need to run a couple of commands to set everything up, results in no-one in the core team being willing to support a production Docker container at this time.

2 Likes

So, the docker image is now merged for a few weeks but there still aren’t any mention of it in the wiki. I would have added it but I’m not sure what to say exactly. Does someone who know better than me want to do it?

See https://github.com/diaspora/diaspora/pull/7870#issuecomment-433735184. And since I am on PTO now, there is a high chance this will be done this year. :wink:

2 Likes

Hello, diaspora* is a the moment incompatible with the last version of Ubuntu which I run on my laptop, and I didn’t think about it but I also upgraded my desktop to the last version of Mint which is now based on the last version of Ubuntu. Long story short: I don’t have any development environment working locally easily now. I think this is the kind of problem docker aims to fix, but if I am not wrong we still have no mention about this docker for development anywhere. Can anybody give me pointer about where it is, and how to use it to develop on a specific branch? Thanks

Just start with script/diaspora-dev --help, so if this is the first time you are using it, you first need to build the container and prepare everything by running script/diaspora-dev setup, and then you can just switch branches and run commands as usual (well, through the script/diaspora-dev script, so the commands run in the container). If your other branch requires different gems, you need to run script/diaspora-dev bundle to update those in the container.

The commands should be pretty self explaining and all have a --help. If something isn’t clear enough, just ask and we can improve the help.

One potential problem: If you have an existing diaspora dev setup, then you probably already have a database.yml, so you either need to change that so it points again to your existing database, and make that reachable from docker, or delete the database.yml, as the docker-setup brings it’s own database, and can create it’s own config for it, but it doesn’t overwrite an existing file. So the script is designed to work out-of-the-box if you clone a fresh repo without configs already existing, but it can also work in other scenarios, but then you need to manually manage your configs.

I finally had time to come back to this, I moved the database,yml config file, and the setup worked, and rspec tests work (well, the one for EXIF photo fails but all the others are fine).

However, cucumber ones don’t run:

rails aborted!
ActiveRecord::NoEnvironmentInSchemaError: 

Environment data not found in the schema. To resolve this issue, run: 

        bin/rails db:environment:set RAILS_ENV=test

/diaspora/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/migration.rb:1173:in `last_stored_environment'
/diaspora/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/tasks/database_tasks.rb:60:in `check_protected_environments!'
/diaspora/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:18:in `block (2 levels) in <top (required)>'
/diaspora/vendor/bundle/ruby/2.7.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'

Well, the solution is right there … but as you need to run this inside docker the full command will look like this:

script/diaspora-dev exec bin/rails db:environment:set RAILS_ENV=test

hm, I’m surprised this isn’t part of the setup command?