15

When I start a new M2 project, the first thing I would do is to install the core via composer:

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition

I can now write my custom module(s) and theme(s) under app/code. I would then add my composer.* and the whole app/code folder to my VCS. So far everything is fine.

Suppose now I want to use some build tools for my project, let's say Grunt or Gulp.

  1. If I commit my own Gruntfile.js, this will be overwritten by the magento/magento2-base package when I run composer install after I cloned the repo.

  2. If I commit my gulpfile.js, I can't really define my dependencies in a package.json, because it would also be overwritten by magento/magento2-base.

  3. If I decide to use Magento's Grunt setup and want to customise it by editing the files under /dev/tools/grunt (e.g. themes.js), I can't because my changes would be overwritten by magento/magento2-base.

My understanding is that you can't really do much in your document root. There are of course plenty of solutions to this problem:

  • I could run a git checkout - right after installation to reset my own files
  • I could store my build files in a dedicated folder, /build for example
  • I could use a different build tool, like Phing, Ant, or Rake (my frontend devs wouldn't be so happy though)
  • I could replace magento/magento2-base with a custom package that has a custom mapping for core files (not really optimal but hey, it's an option)

I personally dislike all of these options, so I would like to know if there is a preferred or better way to achieve what I'm trying to do.

Is anyone having the same problem? How did you solve it? How do you structure your project under VCS?

UPDATE

An extra point related to the project setup. In my experiments I noticed that Magento composer installer has a flag for file overriding:

"extra": {
    "magento-force": "override"
}

It is internally treated as a boolean if I'm not mistaken, so I tried to set it to false to skip overriding. When I run composer install my installation fails because of the file(s) being already present. Basically, if I don't let Magento override my files, I can't install it.

What is the purpose of this flag then? Is it only suppose to perform a check for me? It doesn't make much sense to me to be honest, but maybe someone can shed some light on the subject.

  • I'm curious to see what others post as an answer. Ideally, I think we'd want to keep Magento Core out of our main repo and keep that limited to just the template we create and any custom plugins we add or right. Then at build time, we reference the core + our project repo and build an application artifact from the repositories. This is the method I have been using for M1 recently and I'm wondering if the official recommendation from Magento is to do something similar with M2 now that Composer is fully supported. – Bryan 'BJ' Hoffpauir Jr. Jan 27 '16 at 6:43
  • In newer M2 versions, the Gruntfile.js, gulpfile.js and package.json issue is solved. The issue addressed in this question is still applicable to newer Magento 2 versions when you need to change themes.js, index.php or .htaccess for instance. – 7ochem Jan 2 '18 at 9:51
4

Shortish term, we are looking to separate files that need customization. E.g. if people need to modify index.php, work out how to separate the standard file Magento ships from the need of local customizations. Once achieved, it is possible to have a "one true .gitignore for all projects can use". That is, easy to commit whole project directory to Git with .gitignore of everything that "composer install" will fetch for you (and everything "composer update" will replace when installing a patch or upgrade).

Longer term, the goal is to shorten .gitignore as much as possible. E.g. push more into modules under the 'vendor' directory.

Then

  1. For everything you don't want to share across projects, leave it under app/code and committed in the main project repo.
  2. Everything locally developed you want to share across projects more easily, put in a separate GIT repo and install via composer so it ends up under 'vendor'. (Could be a local composer repo, or just install straight from GIT.)

That way you can still git commit the whole project tree from the top down, capturing the composer.json and composer.lock files (committing just app/code does not). The .gitignore will exclude the 'vendor' directory and other files not wanted.

This gives you the best of both worlds mentioned in the other discussion. The current pain is the length and complexity of the .gitignore file, and patch installation currently wipes out some local customization (e.g. in index.php). Short term workaround - remove index.php from .gitignore, and when you install a patch check to see what changes you lost (git diff) and reapply them manually.

  • OK, so you will change some things there in the near future, nice! I wonder if this "magento-force": "override" flag could be useful somehow. At the moment it is not exactly doing what I would expect. In case you edited/extended your index.php or any other "core" files, you could simply tell Magento not to overwrite your changes. Does it make any sense? – fmrng Jan 29 '16 at 8:18
3

There is an easy Solution for your override Problem: don't change core Files ;) Magento is based on extending the Code and not changing it.

First thing is, you should not put your whole app/code folder in one vcs Repository. Each Magento Component (Module, Theme, etc...) should be a repository itself.

If you want to change/extend the frontend, you should create a new theme and treat this theme as your grunt project, not the whole Magento2 Instance.

To install your theme in your Project you can easily pull it in via composer directly from your vcs repository

  • Well, the app/code folder is specifically there to customise Magento. My understanding of the current M2 is that app/code replaces what app/code/local was in M1, and community modules can be installed via composer under vendor. We have some projects with a LARGE number of modules, and several themes as well. What you are suggesting would be impossible to manage. – fmrng Jan 27 '16 at 9:31
  • Hey, we manage projects with >100 components that way. The key is to keep modules small and manage your composer dependencies between the modules. You can clone the magento project repository for your own needs and add all your components to your project – David Verholen Jan 27 '16 at 9:39
  • If you are happy with your current setup it's fine. Honestly, I find it rather cumbersome. It means you have 100+ git repositories, and every time you are changing something you have to open a specific project, commit your changes, run composer update. Where do you commit your composer.lock then? If you have 10+ devs working on the same project, it could turn out really messy. Of course we do have plenty of general modules (and themes even) that we install via composer, but project-specific code should be versioned under the same repo for the sake of clarity. – fmrng Jan 27 '16 at 9:55
  • I'm not saying you are doing it wrong, I think it's a bit overcomplicated for my taste. Out of interest, how do you inspect your repo history with such a setup? How can you use features like git blame or git log when the code is scattered in multiple components? Do you run integration tests to see that everything is working fine? – fmrng Jan 27 '16 at 10:04
  • we had this discussion internally last year and deployments got rather simple since we decided to make it 1repo=1module. The Point is that you would not make a composer update for one little change. Developers work in dev environments and change files there directly. When they are done, they can tag it as alpha, beta or release candidate. This way, several developers can work on many projects at the same time and the next time, you make a composer update you pull in all changes. There are great tools for organizing your vcs and composer packages.Hundreds of Repositories should not be a Problem – David Verholen Jan 27 '16 at 10:07
2

Ok, looks like I found a better solution for what I was trying to achieve. In the composer.json, it is possible to specify which files should be ignored by the Magento Composer Installer. If I don't want my Gruntfile.js to be overridden, I can simply specify it with the following configuration:

"extra": {
    "magento-deploy-ignore": {
        "magento/magento2-base": [
            "/Gruntfile.js",
            "/package.json"
        ]
    }
}

I am now able to extend the standard installation to fit my needs.

  • This solution doesn't seem "upgrade safe". If Magento will make changes to these files you ignore, then you won't know or you will forget about these files. You are using your own version which will never include these new changes. Please check my answer for my suggestion. – 7ochem Jan 2 '18 at 10:11
2

Unfortunately the accepted answer, although being the way originally intended to achieve the desired goal, only work for excluding files and directories placed in the root, because if we want to exclude a file placed in a subdirectory (eg. dev/tools/grunt/configs/themes.js, needed if we add a new theme and want to use Magento Grunt tasks), putting it in the "magento-deploy-ignore" configuration, it blocks the deploy of all parent directories (that is, dev and all its subdirectories).

This happen because the method that process the "magento-deploy-ignore" (\MagentoHackathon\Composer\Magento\Deploystrategy\DeploystrategyAbstract::isDestinationIgnored) use strpos to match the destination path against the list of excluded, so every parent path will always return true.

  • I know, I could see that in my tests, but since we are using a different build workflow it works fine for us atm. Could you find a better option? – fmrng Jul 22 '16 at 9:26
  • We started doing the checkout of the files during the build phase of our pipeline, then we stopped using at all built-in Grunt tasks so atm it's not a problem. – Gennaro Vietri Jul 23 '16 at 9:59
  • By the way, we started evaluate of fork magento-composer-installer for enhance the "magento-deploy-ignore" behaviour, if the problem should reappear we could follow this path – Gennaro Vietri Jul 23 '16 at 10:02
0

Using patches

What I use is creating and applying patches. When we need to change dev/tools/grunt/configs/themes.js, index.php or .htaccess we apply the changes to a temporary copy of the file and create a patch out of it (create a build/ dir first):

$ cp dev/tools/grunt/configs/themes.js dev/tools/grunt/configs/themes.js.tmp
  # Now Make changes in .tmp file
$ diff -u3 dev/tools/grunt/configs/themes.js dev/tools/grunt/configs/themes.js.tmp | sed 's/\.tmp//' > build/themes.patch
$ mv dev/tools/grunt/configs/themes.js.tmp dev/tools/grunt/configs/themes.js

Then we can make this patch apply automatically when running composer install or update by adding te commands to the scripts section of your composer.json file:

{
    "scripts": {
        "post-install-cmd": "patch -i build/themes.patch dev/tools/grunt/configs/themes.js",
        "post-update-cmd": "patch -i build/themes.patch dev/tools/grunt/configs/themes.js"
    }
}

(Also you could put the above patch ... command in a bash script, let's say build/themes_patch.sh and call that script from Composer so it will be reusable or manually executable)

Upgrade safe! :D

This solution is upgrade safe! You are not changing core files directly without respecting the original file. You are applying a patch to the original Magento2 file. When that file changes because you are upgrading, the patch will fail and you know that you have to take a closer look to the new changes and create a new patch.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.