diff --git a/.coveragerc b/.coveragerc deleted file mode 100644 index d7a24812..00000000 --- a/.coveragerc +++ /dev/null @@ -1,34 +0,0 @@ -[run] -branch = True -source = . -omit = - .tox/* - /usr/* - setup.py - # Don't complain if non-runnable code isn't run - */__main__.py - pre_commit/color_windows.py - pre_commit/resources/* - -[report] -show_missing = True -skip_covered = True -exclude_lines = - # Have to re-enable the standard pragma - \#\s*pragma: no cover - # We optionally substitute this - ${COVERAGE_IGNORE_WINDOWS} - - # Don't complain if tests don't hit defensive assertion code: - ^\s*raise AssertionError\b - ^\s*raise NotImplementedError\b - ^\s*return NotImplemented\b - ^\s*raise$ - - # Don't complain if non-runnable code isn't run: - ^if __name__ == ['"]__main__['"]:$ - -[html] -directory = coverage-html - -# vim:ft=dosini diff --git a/.github/ISSUE_TEMPLATE/00_bug.yaml b/.github/ISSUE_TEMPLATE/00_bug.yaml new file mode 100644 index 00000000..980f7afe --- /dev/null +++ b/.github/ISSUE_TEMPLATE/00_bug.yaml @@ -0,0 +1,54 @@ +name: bug report +description: something went wrong +body: + - type: markdown + attributes: + value: | + this is for issues for `pre-commit` (the framework). + if you are reporting an issue for [pre-commit.ci] please report it at [pre-commit-ci/issues] + + [pre-commit.ci]: https://pre-commit.ci + [pre-commit-ci/issues]: https://github.com/pre-commit-ci/issues + - type: input + id: search + attributes: + label: search you tried in the issue tracker + placeholder: ... + validations: + required: true + - type: markdown + attributes: + value: | + 95% of issues created are duplicates. + please try extra hard to find them first. + it's very unlikely your problem is unique. + - type: textarea + id: freeform + attributes: + label: describe your issue + placeholder: 'I was doing ... I ran ... I expected ... I got ...' + validations: + required: true + - type: input + id: version + attributes: + label: pre-commit --version + placeholder: pre-commit x.x.x + validations: + required: true + - type: textarea + id: configuration + attributes: + label: .pre-commit-config.yaml + description: (auto-rendered as yaml, no need for backticks) + placeholder: 'repos: ...' + render: yaml + validations: + required: true + - type: textarea + id: error-log + attributes: + label: '~/.cache/pre-commit/pre-commit.log (if present)' + placeholder: "### version information\n..." + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/01_feature.yaml b/.github/ISSUE_TEMPLATE/01_feature.yaml new file mode 100644 index 00000000..c7ddc84c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/01_feature.yaml @@ -0,0 +1,38 @@ +name: feature request +description: something new +body: + - type: markdown + attributes: + value: | + this is for issues for `pre-commit` (the framework). + if you are reporting an issue for [pre-commit.ci] please report it at [pre-commit-ci/issues] + + [pre-commit.ci]: https://pre-commit.ci + [pre-commit-ci/issues]: https://github.com/pre-commit-ci/issues + - type: input + id: search + attributes: + label: search you tried in the issue tracker + placeholder: ... + validations: + required: true + - type: markdown + attributes: + value: | + 95% of issues created are duplicates. + please try extra hard to find them first. + it's very unlikely your feature idea is a new one. + - type: textarea + id: freeform + attributes: + label: describe your actual problem + placeholder: 'I want to do ... I tried ... It does not work because ...' + validations: + required: true + - type: input + id: version + attributes: + label: pre-commit --version + placeholder: pre-commit x.x.x + validations: + required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..4179f47f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: +- name: documentation + url: https://pre-commit.com + about: please check the docs first +- name: pre-commit.ci issues + url: https://github.com/pre-commit-ci/issues + about: please report issues about pre-commit.ci here diff --git a/.github/actions/pre-test/action.yml b/.github/actions/pre-test/action.yml new file mode 100644 index 00000000..b70c942f --- /dev/null +++ b/.github/actions/pre-test/action.yml @@ -0,0 +1,9 @@ +inputs: + env: + default: ${{ matrix.env }} + +runs: + using: composite + steps: + - uses: asottile/workflows/.github/actions/latest-git@v1.4.0 + if: inputs.env == 'py39' && runner.os == 'Linux' diff --git a/.github/workflows/languages.yaml b/.github/workflows/languages.yaml new file mode 100644 index 00000000..be8963ba --- /dev/null +++ b/.github/workflows/languages.yaml @@ -0,0 +1,84 @@ +name: languages + +on: + push: + branches: [main, test-me-*] + tags: '*' + pull_request: + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + vars: + runs-on: ubuntu-latest + outputs: + languages: ${{ steps.vars.outputs.languages }} + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + - uses: actions/setup-python@v4 + with: + python-version: '3.10' + - name: install deps + run: python -mpip install -e . -r requirements-dev.txt + - name: vars + run: testing/languages ${{ github.event_name == 'push' && '--all' || '' }} + id: vars + language: + needs: [vars] + runs-on: ${{ matrix.os }} + if: needs.vars.outputs.languages != '[]' + strategy: + fail-fast: false + matrix: + include: ${{ fromJSON(needs.vars.outputs.languages) }} + steps: + - uses: asottile/workflows/.github/actions/fast-checkout@v1.8.1 + - uses: actions/setup-python@v4 + with: + python-version: '3.10' + + - run: echo "$CONDA\Scripts" >> "$GITHUB_PATH" + shell: bash + if: matrix.os == 'windows-latest' && matrix.language == 'conda' + - run: testing/get-coursier.sh + shell: bash + if: matrix.language == 'coursier' + - run: testing/get-dart.sh + shell: bash + if: matrix.language == 'dart' + - run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends \ + lua5.3 \ + liblua5.3-dev \ + luarocks + if: matrix.os == 'ubuntu-latest' && matrix.language == 'lua' + - run: | + echo 'C:\Strawberry\perl\bin' >> "$GITHUB_PATH" + echo 'C:\Strawberry\perl\site\bin' >> "$GITHUB_PATH" + echo 'C:\Strawberry\c\bin' >> "$GITHUB_PATH" + shell: bash + if: matrix.os == 'windows-latest' && matrix.language == 'perl' + - uses: haskell/actions/setup@v2 + if: matrix.language == 'haskell' + - uses: r-lib/actions/setup-r@v2 + if: matrix.os == 'ubuntu-latest' && matrix.language == 'r' + + - name: install deps + run: python -mpip install -e . -r requirements-dev.txt + - name: run tests + run: coverage run -m pytest tests/languages/${{ matrix.language }}_test.py + - name: check coverage + run: coverage report --include pre_commit/languages/${{ matrix.language }}.py,tests/languages/${{ matrix.language }}_test.py + collector: + needs: [language] + if: always() + runs-on: ubuntu-latest + steps: + - name: check for failures + if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') + run: echo job failed && exit 1 diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml new file mode 100644 index 00000000..02b11ae2 --- /dev/null +++ b/.github/workflows/main.yml @@ -0,0 +1,23 @@ +name: main + +on: + push: + branches: [main, test-me-*] + tags: '*' + pull_request: + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + main-windows: + uses: asottile/workflows/.github/workflows/tox.yml@v1.8.1 + with: + env: '["py310"]' + os: windows-latest + main-linux: + uses: asottile/workflows/.github/workflows/tox.yml@v1.8.1 + with: + env: '["py310", "py311", "py312", "py313"]' + os: ubuntu-latest diff --git a/.gitignore b/.gitignore index ae552f4a..c2021816 100644 --- a/.gitignore +++ b/.gitignore @@ -1,14 +1,6 @@ *.egg-info -*.iml *.py[co] -.*.sw[a-z] -.coverage -.idea -.project -.pydevproject -.tox -.venv.touch -/venv* -coverage-html -dist -.pytest_cache +/.coverage +/.tox +/dist +.vscode/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1b87a406..3654066f 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,42 +1,44 @@ repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v2.1.0 + rev: v6.0.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - - id: check-docstring-first - - id: check-json - id: check-yaml - id: debug-statements + - id: double-quote-string-fixer - id: name-tests-test - id: requirements-txt-fixer - - id: double-quote-string-fixer -- repo: https://gitlab.com/pycqa/flake8 - rev: 3.7.7 +- repo: https://github.com/asottile/setup-cfg-fmt + rev: v3.2.0 hooks: - - id: flake8 -- repo: https://github.com/pre-commit/mirrors-autopep8 - rev: v1.4.3 - hooks: - - id: autopep8 -- repo: https://github.com/pre-commit/pre-commit - rev: v1.14.4 - hooks: - - id: validate_manifest -- repo: https://github.com/asottile/pyupgrade - rev: v1.12.0 - hooks: - - id: pyupgrade -- repo: https://github.com/asottile/reorder_python_imports - rev: v1.4.0 + - id: setup-cfg-fmt +- repo: https://github.com/asottile/reorder-python-imports + rev: v3.16.0 hooks: - id: reorder-python-imports - language_version: python3 + exclude: ^pre_commit/resources/ + args: [--py310-plus, --add-import, 'from __future__ import annotations'] - repo: https://github.com/asottile/add-trailing-comma - rev: v1.0.0 + rev: v4.0.0 hooks: - id: add-trailing-comma -- repo: meta +- repo: https://github.com/asottile/pyupgrade + rev: v3.21.2 hooks: - - id: check-hooks-apply - - id: check-useless-excludes + - id: pyupgrade + args: [--py310-plus] +- repo: https://github.com/hhatto/autopep8 + rev: v2.3.2 + hooks: + - id: autopep8 +- repo: https://github.com/PyCQA/flake8 + rev: 7.3.0 + hooks: + - id: flake8 +- repo: https://github.com/pre-commit/mirrors-mypy + rev: v1.19.1 + hooks: + - id: mypy + additional_dependencies: [types-pyyaml] + exclude: ^testing/resources/ diff --git a/.pre-commit-hooks.yaml b/.pre-commit-hooks.yaml index ef269d13..e1aaf583 100644 --- a/.pre-commit-hooks.yaml +++ b/.pre-commit-hooks.yaml @@ -1,6 +1,6 @@ - id: validate_manifest - name: Validate Pre-Commit Manifest + name: validate pre-commit manifest description: This validator validates a pre-commit hooks manifest file - entry: pre-commit-validate-manifest + entry: pre-commit validate-manifest language: python - files: ^(\.pre-commit-hooks\.yaml|hooks\.yaml)$ + files: ^\.pre-commit-hooks\.yaml$ diff --git a/CHANGELOG.md b/CHANGELOG.md index ba491cfb..879ae073 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,1226 @@ +4.5.1 - 2025-12-16 +================== + +### Fixes +- Fix `language: python` with `repo: local` without `additional_dependencies`. + - #3597 PR by @asottile. + +4.5.0 - 2025-11-22 +================== + +### Features +- Add `pre-commit hazmat`. + - #3585 PR by @asottile. + +4.4.0 - 2025-11-08 +================== + +### Features +- Add `--fail-fast` option to `pre-commit run`. + - #3528 PR by @JulianMaurin. +- Upgrade `ruby-build` / `rbenv`. + - #3566 PR by @asottile. + - #3565 issue by @MRigal. +- Add `language: unsupported` / `language: unsupported_script` as aliases + for `language: system` / `language: script` (which will eventually be + deprecated). + - #3577 PR by @asottile. +- Add support docker-in-docker detection for cgroups v2. + - #3535 PR by @br-rhrbacek. + - #3360 issue by @JasonAlt. + +### Fixes +- Handle when docker gives `SecurityOptions: null`. + - #3537 PR by @asottile. + - #3514 issue by @jenstroeger. +- Fix error context for invalid `stages` in `.pre-commit-config.yaml`. + - #3576 PR by @asottile. + +4.3.0 - 2025-08-09 +================== + +### Features +- `language: docker` / `language: docker_image`: detect rootless docker. + - #3446 PR by @matthewhughes934. + - #1243 issue by @dkolepp. +- `language: julia`: avoid `startup.jl` when executing hooks. + - #3496 PR by @ericphanson. +- `language: dart`: support latest dart versions which require a higher sdk + lower bound. + - #3507 PR by @bc-lee. + +4.2.0 - 2025-03-18 +================== + +### Features +- For `language: python` first attempt a versioned python executable for + the default language version before consulting a potentially unversioned + `sys.executable`. + - #3430 PR by @asottile. + +### Fixes +- Handle error during conflict detection when a file is named "HEAD" + - #3425 PR by @tusharsadhwani. + +4.1.0 - 2025-01-20 +================== + +### Features +- Add `language: julia`. + - #3348 PR by @fredrikekre. + - #2689 issue @jmuchovej. + +### Fixes +- Disable automatic toolchain switching for `language: golang`. + - #3304 PR by @AleksaC. + - #3300 issue by @AleksaC. + - #3149 issue by @nijel. +- Fix `language: r` installation when initiated by RStudio. + - #3389 PR by @lorenzwalthert. + - #3385 issue by @lorenzwalthert. + + +4.0.1 - 2024-10-08 +================== + +### Fixes +- Fix `pre-commit migrate-config` for unquoted deprecated stages names with + purelib `pyyaml`. + - #3324 PR by @asottile. + - pre-commit-ci/issues#234 issue by @lorenzwalthert. + +4.0.0 - 2024-10-05 +================== + +### Features +- Improve `pre-commit migrate-config` to handle more yaml formats. + - #3301 PR by @asottile. +- Handle `stages` deprecation in `pre-commit migrate-config`. + - #3302 PR by @asottile. + - #2732 issue by @asottile. +- Upgrade `ruby-build`. + - #3199 PR by @ThisGuyCodes. +- Add "sensible regex" warnings to `repo: meta`. + - #3311 PR by @asottile. +- Add warnings for deprecated `stages` (`commit` -> `pre-commit`, `push` -> + `pre-push`, `merge-commit` -> `pre-merge-commit`). + - #3312 PR by @asottile. + - #3313 PR by @asottile. + - #3315 PR by @asottile. + - #2732 issue by @asottile. + +### Updating +- `language: python_venv` has been removed -- use `language: python` instead. + - #3320 PR by @asottile. + - #2734 issue by @asottile. + +3.8.0 - 2024-07-28 +================== + +### Features +- Implement health checks for `language: r` so environments are recreated if + the system version of R changes. + - #3206 issue by @lorenzwalthert. + - #3265 PR by @lorenzwalthert. + +3.7.1 - 2024-05-10 +================== + +### Fixes +- Fix `language: rust` default language version check when `rust-toolchain.toml` + is present. + - issue by @gaborbernat. + - #3201 PR by @asottile. + +3.7.0 - 2024-03-24 +================== + +### Features +- Use a tty for `docker` and `docker_image` hooks when `--color` is specified. + - #3122 PR by @glehmann. + +### Fixes +- Fix `fail_fast` for individual hooks stopping when previous hooks had failed. + - #3167 issue by @tp832944. + - #3168 PR by @asottile. + +### Updating +- The per-hook behaviour of `fail_fast` was fixed. If you want the pre-3.7.0 + behaviour, add `fail_fast: true` to all hooks before the last `fail_fast` + hook. + +3.6.2 - 2024-02-18 +================== + +### Fixes +- Fix building golang hooks during `git commit --all`. + - #3130 PR by @asottile. + - #2722 issue by @pestanko and @matthewhughes934. + +3.6.1 - 2024-02-10 +================== + +### Fixes +- Remove `PYTHONEXECUTABLE` from environment when running. + - #3110 PR by @untitaker. +- Handle staged-files-only with only a crlf diff. + - #3126 PR by @asottile. + - issue by @tyyrok. + +3.6.0 - 2023-12-09 +================== + +### Features +- Check `minimum_pre_commit_version` first when parsing configs. + - #3092 PR by @asottile. + +### Fixes +- Fix deprecation warnings for `importlib.resources`. + - #3043 PR by @asottile. +- Fix deprecation warnings for rmtree. + - #3079 PR by @edgarrmondragon. + +### Updating +- Drop support for python<3.9. + - #3042 PR by @asottile. + - #3093 PR by @asottile. + +3.5.0 - 2023-10-13 +================== + +### Features +- Improve performance of `check-hooks-apply` and `check-useless-excludes`. + - #2998 PR by @mxr. + - #2935 issue by @mxr. + +### Fixes +- Use `time.monotonic()` for more accurate hook timing. + - #3024 PR by @adamchainz. + +### Updating +- Require npm 6.x+ for `language: node` hooks. + - #2996 PR by @RoelAdriaans. + - #1983 issue by @henryiii. + +3.4.0 - 2023-09-02 +================== + +### Features +- Add `language: haskell`. + - #2932 by @alunduil. +- Improve cpu count detection when run under cgroups. + - #2979 PR by @jdb8. + - #2978 issue by @jdb8. + +### Fixes +- Handle negative exit codes from hooks receiving posix signals. + - #2971 PR by @chriskuehl. + - #2970 issue by @chriskuehl. + +3.3.3 - 2023-06-13 +================== + +### Fixes +- Work around OS packagers setting `--install-dir` / `--bin-dir` in gem settings. + - #2905 PR by @jaysoffian. + - #2799 issue by @lmilbaum. + +3.3.2 - 2023-05-17 +================== + +### Fixes +- Work around `r` on windows sometimes double-un-quoting arguments. + - #2885 PR by @lorenzwalthert. + - #2870 issue by @lorenzwalthert. + +3.3.1 - 2023-05-02 +================== + +### Fixes +- Work around `git` partial clone bug for `autoupdate` on windows. + - #2866 PR by @asottile. + - #2865 issue by @adehad. + +3.3.0 - 2023-05-01 +================== + +### Features +- Upgrade ruby-build. + - #2846 PR by @jalessio. +- Use blobless clone for faster autoupdate. + - #2859 PR by @asottile. +- Add `-j` / `--jobs` argument to `autoupdate` for parallel execution. + - #2863 PR by @asottile. + - issue by @gaborbernat. + +3.2.2 - 2023-04-03 +================== + +### Fixes +- Fix support for swift >= 5.8. + - #2836 PR by @edelabar. + - #2835 issue by @kgrobelny-intive. + +3.2.1 - 2023-03-25 +================== + +### Fixes +- Fix `language_version` for `language: rust` without global `rustup`. + - #2823 issue by @daschuer. + - #2827 PR by @asottile. + +3.2.0 - 2023-03-17 +================== + +### Features +- Allow `pre-commit`, `pre-push`, and `pre-merge-commit` as `stages`. + - #2732 issue by @asottile. + - #2808 PR by @asottile. +- Add `pre-rebase` hook support. + - #2582 issue by @BrutalSimplicity. + - #2725 PR by @mgaligniana. + +### Fixes +- Remove bulky cargo cache from `language: rust` installs. + - #2820 PR by @asottile. + +3.1.1 - 2023-02-27 +================== + +### Fixes +- Fix `rust` with `language_version` and a non-writable host `RUSTUP_HOME`. + - pre-commit-ci/issues#173 by @Swiftb0y. + - #2788 by @asottile. + +3.1.0 - 2023-02-22 +================== + +### Fixes +- Fix `dotnet` for `.sln`-based hooks for dotnet>=7.0.200. + - #2763 PR by @m-rsha. +- Prevent stashing when `diff` fails to execute. + - #2774 PR by @asottile. + - #2773 issue by @strubbly. +- Dependencies are no longer sorted in repository key. + - #2776 PR by @asottile. + +### Updating +- Deprecate `language: python_venv`. Use `language: python` instead. + - #2746 PR by @asottile. + - #2734 issue by @asottile. + + +3.0.4 - 2023-02-03 +================== + +### Fixes +- Fix hook diff detection for files affected by `--textconv`. + - #2743 PR by @adamchainz. + - #2743 issue by @adamchainz. + +3.0.3 - 2023-02-01 +================== + +### Fixes +- Revert "Prevent local `Gemfile` from interfering with hook execution.". + - #2739 issue by @Roguelazer. + - #2740 PR by @asottile. + +3.0.2 - 2023-01-29 +================== + +### Fixes +- Prevent local `Gemfile` from interfering with hook execution. + - #2727 PR by @asottile. +- Fix `language: r`, `repo: local` hooks + - pre-commit-ci/issues#107 by @lorenzwalthert. + - #2728 PR by @asottile. + +3.0.1 - 2023-01-26 +================== + +### Fixes +- Ensure coursier hooks are available offline after install. + - #2723 PR by @asottile. + +3.0.0 - 2023-01-23 +================== + +### Features +- Make `language: golang` bootstrap `go` if not present. + - #2651 PR by @taoufik07. + - #2649 issue by @taoufik07. +- `language: coursier` now supports `additional_dependencies` and `repo: local` + - #2702 PR by @asottile. +- Upgrade `ruby-build` to `20221225`. + - #2718 PR by @jalessio. + +### Fixes +- Improve error message for invalid yaml for `pre-commit autoupdate`. + - #2686 PR by @asottile. + - #2685 issue by @CarstenGrohmann. +- `repo: local` no longer provisions an empty `git` repo. + - #2699 PR by @asottile. + +### Updating +- Drop support for python<3.8 + - #2655 PR by @asottile. +- Drop support for top-level list, use `pre-commit migrate-config` to update. + - #2656 PR by @asottile. +- Drop support for `sha` to specify revision, use `pre-commit migrate-config` + to update. + - #2657 PR by @asottile. +- Remove `pre-commit-validate-config` and `pre-commit-validate-manifest`, use + `pre-commit validate-config` and `pre-commit validate-manifest` instead. + - #2658 PR by @asottile. +- `language: golang` hooks must use `go.mod` to specify dependencies + - #2672 PR by @taoufik07. + + +2.21.0 - 2022-12-25 +=================== + +### Features +- Require new-enough virtualenv to prevent 3.10 breakage + - #2467 PR by @asottile. +- Respect aliases with `SKIP` for environment install. + - #2480 PR by @kmARC. + - #2478 issue by @kmARC. +- Allow `pre-commit run --files` against unmerged paths. + - #2484 PR by @asottile. +- Also apply regex warnings to `repo: local` hooks. + - #2524 PR by @chrisRedwine. + - #2521 issue by @asottile. +- `rust` is now a "first class" language -- supporting `language_version` and + installation when not present. + - #2534 PR by @Holzhaus. +- `r` now uses more-reliable binary installation. + - #2460 PR by @lorenzwalthert. +- `GIT_ALLOW_PROTOCOL` is now passed through for git operations. + - #2555 PR by @asottile. +- `GIT_ASKPASS` is now passed through for git operations. + - #2564 PR by @mattp-. +- Remove `toml` dependency by using `cargo add` directly. + - #2568 PR by @m-rsha. +- Support `dotnet` hooks which have dotted prefixes. + - #2641 PR by @rkm. + - #2629 issue by @rkm. + +### Fixes +- Properly adjust `--commit-msg-filename` if run from a sub directory. + - #2459 PR by @asottile. +- Simplify `--intent-to-add` detection by using `git diff`. + - #2580 PR by @m-rsha. +- Fix `R.exe` selection on windows. + - #2605 PR by @lorenzwalthert. + - #2599 issue by @SInginc. +- Skip default `nuget` source when installing `dotnet` packages. + - #2642 PR by @rkm. + +2.20.0 - 2022-07-10 +=================== + +### Features +- Expose `source` and `object-name` (positional args) of `prepare-commit-msg` + hook as `PRE_COMMIT_COMIT_MSG_SOURCE` and `PRE_COMMIT_COMMIT_OBJECT_NAME`. + - #2407 PR by @M-Whitaker. + - #2406 issue by @M-Whitaker. + +### Fixes +- Fix `language: ruby` installs when `--user-install` is set in gemrc. + - #2394 PR by @narpfel. + - #2393 issue by @narpfel. +- Adjust pty setup for solaris. + - #2390 PR by @gaige. + - #2389 issue by @gaige. +- Remove unused `--config` option from `gc`, `sample-config`, + `validate-config`, `validate-manifest` sub-commands. + - #2429 PR by @asottile. + +2.19.0 - 2022-05-05 +=================== + +### Features +- Allow multiple outputs from `language: dotnet` hooks. + - #2332 PR by @WallucePinkham. +- Add more information to `healthy()` failure. + - #2348 PR by @asottile. +- Upgrade ruby-build. + - #2342 PR by @jalessio. +- Add `pre-commit validate-config` / `pre-commit validate-manifest` and + deprecate `pre-commit-validate-config` and `pre-commit-validate-manifest`. + - #2362 PR by @asottile. + +### Fixes +- Fix `pre-push` when pushed ref contains spaces. + - #2345 PR by @wwade. + - #2344 issue by @wwade. + +### Updating +- Change `pre-commit-validate-config` / `pre-commit-validate-manifest` to + `pre-commit validate-config` / `pre-commit validate-manifest`. + - #2362 PR by @asottile. + +2.18.1 - 2022-04-02 +=================== + +### Fixes +- Fix regression for `repo: local` hooks running `python<3.7` + - #2324 PR by @asottile. + +2.18.0 - 2022-04-02 +=================== + +### Features +- Keep `GIT_HTTP_PROXY_AUTHMETHOD` in git environ. + - #2272 PR by @VincentBerthier. + - #2271 issue by @VincentBerthier. +- Support both `cs` and `coursier` executables for coursier hooks. + - #2293 PR by @Holzhaus. +- Include more information in errors for `language_version` / + `additional_dependencies` for languages which do not support them. + - #2315 PR by @asottile. +- Have autoupdate preferentially pick tags which look like versions when + there are multiple equivalent tags. + - #2312 PR by @mblayman. + - #2311 issue by @mblayman. +- Upgrade `ruby-build`. + - #2319 PR by @jalessio. +- Add top level `default_install_hook_types` which will be installed when + `--hook-types` is not specified in `pre-commit install`. + - #2322 PR by @asottile. + +### Fixes +- Fix typo in help message for `--from-ref` and `--to-ref`. + - #2266 PR by @leetrout. +- Prioritize binary builds for R dependencies. + - #2277 PR by @lorenzwalthert. +- Fix handling of git worktrees. + - #2252 PR by @daschuer. +- Fix handling of `$R_HOME` for R hooks. + - #2301 PR by @jeff-m-sullivan. + - #2300 issue by @jeff-m-sullivan. +- Fix a rare race condition in change stashing. + - #2323 PR by @asottile. + - #2287 issue by @ian-h-chamberlain. + +### Updating +- Remove python3.6 support. Note that pre-commit still supports running hooks + written in older versions, but pre-commit itself requires python 3.7+. + - #2215 PR by @asottile. +- pre-commit has migrated from the `master` branch to `main`. + - #2302 PR by @asottile. + +2.17.0 - 2022-01-18 +=================== + +### Features +- add warnings for regexes containing `[\\/]`. + - #2151 issue by @sanjioh. + - #2154 PR by @kuviokelluja. +- upgrade supported ruby versions. + - #2205 PR by @jalessio. +- allow `language: conda` to use `mamba` or `micromamba` via + `PRE_COMMIT_USE_MAMBA=1` or `PRE_COMMIT_USE_MICROMAMBA=1` respectively. + - #2204 issue by @janjagusch. + - #2207 PR by @xhochy. +- display `git --version` in error report. + - #2210 PR by @asottile. +- add `language: lua` as a supported language. + - #2158 PR by @mblayman. + +### Fixes +- temporarily add `setuptools` to the zipapp. + - #2122 issue by @andreoliwa. + - a737d5f commit by @asottile. +- use `go install` instead of `go get` for go 1.18+ support. + - #2161 PR by @schmir. +- fix `language: r` with a local renv and `RENV_PROJECT` set. + - #2170 PR by @lorenzwalthert. +- forbid overriding `entry` in `language: meta` hooks which breaks them. + - #2180 issue by @DanKaplanSES. + - #2181 PR by @asottile. +- always use `#!/bin/sh` on windows for hook script. + - #2182 issue by @hushigome-visco. + - #2187 PR by @asottile. + +2.16.0 - 2021-11-30 +=================== + +### Features +- add warning for regexes containing `[\/]` or `[/\\]`. + - #2053 PR by @radek-sprta. + - #2043 issue by @asottile. +- move hook template back to `bash` resolving shebang-portability issues. + - #2065 PR by @asottile. +- add support for `fail_fast` at the individual hook level. + - #2097 PR by @colens3. + - #1143 issue by @potiuk. +- allow passthrough of `GIT_CONFIG_KEY_*`, `GIT_CONFIG_VALUE_*`, and + `GIT_CONFIG_COUNT`. + - #2136 PR by @emzeat. + +### Fixes +- fix pre-commit autoupdate for `core.useBuiltinFSMonitor=true` on windows. + - #2047 PR by @asottile. + - #2046 issue by @lcnittl. +- fix temporary file stashing with for `submodule.recurse=1`. + - #2071 PR by @asottile. + - #2063 issue by @a666. +- ban broken importlib-resources versions. + - #2098 PR by @asottile. +- replace `exit(...)` with `raise SystemExit(...)` for portability. + - #2103 PR by @asottile. + - #2104 PR by @asottile. + + +2.15.0 - 2021-09-02 +=================== + +### Features +- add support for hooks written in `dart`. + - #2027 PR by @asottile. +- add support for `post-rewrite` hooks. + - #2036 PR by @uSpike. + - #2035 issue by @uSpike. + +### Fixes +- fix `check-useless-excludes` with exclude matching broken symlink. + - #2029 PR by @asottile. + - #2019 issue by @pkoch. +- eliminate duplicate mutable sha warning messages for `pre-commit autoupdate`. + - #2030 PR by @asottile. + - #2010 issue by @graingert. + +2.14.1 - 2021-08-28 +=================== + +### Fixes +- fix force-push of disparate histories using git>=2.28. + - #2005 PR by @asottile. + - #2002 issue by @bogusfocused. +- fix `check-useless-excludes` and `check-hooks-apply` matching non-root + `.pre-commit-config.yaml`. + - #2026 PR by @asottile. + - pre-commit-ci/issues#84 issue by @billsioros. + +2.14.0 - 2021-08-06 +=================== + +### Features +- During `pre-push` hooks, expose local branch as `PRE_COMMIT_LOCAL_BRANCH`. + - #1947 PR by @FlorentClarret. + - #1410 issue by @MaicoTimmerman. +- Improve container id detection for docker-beside-docker with custom hostname. + - #1919 PR by @adarnimrod. + - #1918 issue by @adarnimrod. + +### Fixes +- Read legacy hooks in an encoding-agnostic way. + - #1943 PR by @asottile. + - #1942 issue by @sbienkow-ninja. +- Fix execution of docker hooks for docker-in-docker. + - #1997 PR by @asottile. + - #1978 issue by @robin-moss. + +2.13.0 - 2021-05-21 +=================== + +### Features +- Setting `SKIP=...` skips installation as well. + - #1875 PR by @asottile. + - pre-commit-ci/issues#53 issue by @TylerYep. +- Attempt to mount from host with docker-in-docker. + - #1888 PR by @okainov. + - #1387 issue by @okainov. +- Enable `repo: local` for `r` hooks. + - #1878 PR by @lorenzwalthert. +- Upgrade `ruby-build` and `rbenv`. + - #1913 PR by @jalessio. + +### Fixes +- Better detect `r` packages. + - #1898 PR by @lorenzwalthert. +- Avoid warnings with mismatched `renv` versions. + - #1841 PR by @lorenzwalthert. +- Reproducibly produce ruby tar resources. + - #1915 PR by @asottile. + +2.12.1 - 2021-04-16 +=================== + +### Fixes +- Fix race condition when stashing files in multiple parallel invocations + - #1881 PR by @adamchainz. + - #1880 issue by @adamchainz. + +2.12.0 - 2021-04-06 +=================== + +### Features +- Upgrade rbenv. + - #1854 PR by @asottile. + - #1848 issue by @sirosen. + +### Fixes +- Give command length a little more room when running batch files on windows + so underlying commands can expand further. + - #1864 PR by @asottile. + - pre-commit/mirrors-prettier#7 issue by @DeltaXWizard. +- Fix permissions of root folder in ruby archives. + - #1868 PR by @asottile. + +2.11.1 - 2021-03-09 +=================== + +### Fixes +- Fix r hooks when hook repo is a package + - #1831 PR by @lorenzwalthert. + +2.11.0 - 2021-03-07 +=================== + +### Features +- Improve warning for mutable ref. + - #1809 PR by @JamMarHer. +- Add support for `post-merge` hook. + - #1800 PR by @psacawa. + - #1762 issue by @psacawa. +- Add `r` as a supported hook language. + - #1799 PR by @lorenzwalthert. + +### Fixes +- Fix `pre-commit install` on `subst` / network drives on windows. + - #1814 PR by @asottile. + - #1802 issue by @goroderickgo. +- Fix installation of `local` golang repositories for go 1.16. + - #1818 PR by @rafikdraoui. + - #1815 issue by @rafikdraoui. + +2.10.1 - 2021-02-06 +=================== + +### Fixes +- Fix `language: golang` repositories containing recursive submodules + - #1788 issue by @gaurav517. + - #1789 PR by @paulhfischer. + +2.10.0 - 2021-01-27 +=================== + +### Features +- Allow `ci` as a top-level map for configuration for https://pre-commit.ci + - #1735 PR by @asottile. +- Add warning for mutable `rev` in configuration + - #1715 PR by @paulhfischer. + - #974 issue by @asottile. +- Add warning for `/*` in top-level `files` / `exclude` regexes + - #1750 PR by @paulhfischer. + - #1702 issue by @asottile. +- Expose `PRE_COMMIT_REMOTE_BRANCH` environment variable during `pre-push` + hooks + - #1770 PR by @surafelabebe. +- Produce error message for `language` / `language_version` for non-installable + languages + - #1771 PR by @asottile. + +### Fixes +- Fix execution in worktrees in subdirectories of bare repositories + - #1778 PR by @asottile. + - #1777 issue by @s0undt3ch. + +2.9.3 - 2020-12-07 +================== + +### Fixes +- Fix crash on cygwin mismatch check outside of a git directory + - #1721 PR by @asottile. + - #1720 issue by @chronoB. +- Fix cleanup code on docker volumes for go + - #1725 PR by @fsouza. +- Fix working directory detection on SUBST drives on windows + - #1727 PR by @mrogaski. + - #1610 issue by @jcameron73. + +2.9.2 - 2020-11-25 +================== + +### Fixes +- Fix default value for `types_or` so `symlink` and `directory` can be matched + - #1716 PR by @asottile. + - #1718 issue by @CodeBleu. + +2.9.1 - 2020-11-25 +================== + +### Fixes +- Improve error message for "hook goes missing" + - #1709 PR by @paulhfischer. + - #1708 issue by @theod07. +- Add warning for `/*` in `files` / `exclude` regexes + - #1707 PR by @paulhfischer. + - #1702 issue by @asottile. +- Fix `healthy()` check for `language: python` on windows when the base + executable has non-ascii characters. + - #1713 PR by @asottile. + - #1711 issue by @Najiva. + +2.9.0 - 2020-11-21 +================== + +### Features +- Add `types_or` which allows matching multiple disparate `types` in a hook + - #1677 by @MarcoGorelli. + - #607 by @asottile. +- Add Github Sponsors / Open Collective links + - https://github.com/sponsors/asottile + - https://opencollective.com/pre-commit + +### Fixes +- Improve cleanup for `language: dotnet` + - #1678 by @rkm. +- Fix "xargs" when running windows batch files + - #1686 PR by @asottile. + - #1604 issue by @apietrzak. + - #1604 issue by @ufwtlsb. +- Fix conflict with external `rbenv` and `language_version: default` + - #1700 PR by @asottile. + - #1699 issue by @abuxton. +- Improve performance of `git status` / `git diff` commands by ignoring + submodules + - #1704 PR by @Vynce. + - #1701 issue by @Vynce. + +2.8.2 - 2020-10-30 +================== + +### Fixes +- Fix installation of ruby hooks with `language_version: default` + - #1671 issue by @aerickson. + - #1672 PR by @asottile. + +2.8.1 - 2020-10-28 +================== + +### Fixes +- Allow default `language_version` of `system` when the homedir is `/` + - #1669 PR by @asottile. + +2.8.0 - 2020-10-28 +================== + +### Features +- Update `rbenv` / `ruby-build` + - #1612 issue by @tdeo. + - #1614 PR by @asottile. +- Update `sample-config` versions + - #1611 PR by @mcsitter. +- Add new language: `dotnet` + - #1598 by @rkm. +- Add `--negate` option to `language: pygrep` hooks + - #1643 PR by @MarcoGorelli. +- Add zipapp support + - #1616 PR by @asottile. +- Run pre-commit through https://pre-commit.ci + - #1662 PR by @asottile. +- Add new language: `coursier` (a jvm-based package manager) + - #1633 PR by @JosephMoniz. +- Exit with distinct codes: 1 (user error), 3 (unexpected error), 130 (^C) + - #1601 PR by @int3l. + +### Fixes +- Improve `healthy()` check for `language: node` + `language_version: system` + hooks when the system executable goes missing. + - pre-commit/action#45 issue by @KOliver94. + - #1589 issue by @asottile. + - #1590 PR by @asottile. +- Fix excess whitespace in error log traceback + - #1592 PR by @asottile. +- Fix posixlike shebang invocations with shim executables of the git hook + script on windows. + - #1593 issue by @Celeborn2BeAlive. + - #1595 PR by @Celeborn2BeAlive. +- Remove hard-coded `C:\PythonXX\python.exe` path on windows as it caused + confusion (and `virtualenv` can sometimes do better) + - #1599 PR by @asottile. +- Fix `language: ruby` hooks when `--format-executable` is present in a gemrc + - issue by `Rainbow Tux` (discord). + - #1603 PR by @asottile. +- Move `cygwin` / `win32` mismatch error earlier to catch msys2 mismatches + - #1605 issue by @danyeaw. + - #1606 PR by @asottile. +- Remove `-p` workaround for old `virtualenv` + - #1617 PR by @asottile. +- Fix `language: node` installations to not symlink outside of the environment + - pre-commit-ci/issues#2 issue by @DanielJSottile. + - #1667 PR by @asottile. +- Don't identify shim executables as valid `system` for defaulting + `language_version` for `language: node` / `language: ruby` + - #1658 issue by @adithyabsk. + - #1668 PR by @asottile. + + +2.7.1 - 2020-08-23 +================== + +### Fixes +- Improve performance of docker hooks by removing slow `ps` call + - #1572 PR by @rkm. + - #1569 issue by @asottile. +- Fix un-`healthy()` invalidation followed by install being reported as + un-`healthy()`. + - #1576 PR by @asottile. + - #1575 issue by @jab. +- Fix rare file race condition on windows with `os.replace()` + - #1577 PR by @asottile. + +2.7.0 - 2020-08-22 +================== + +### Features +- Produce error message if an environment is immediately unhealthy + - #1535 PR by @asottile. +- Add --no-allow-missing-config option to init-templatedir + - #1539 PR by @singergr. +- Add warning for old list-style configuration + - #1544 PR by @asottile. +- Allow pre-commit to succeed on a readonly store. + - #1570 PR by @asottile. + - #1536 issue by @asottile. + +### Fixes +- Fix error messaging when the store directory is readonly + - #1546 PR by @asottile. + - #1536 issue by @asottile. +- Improve `diff` performance with many hooks + - #1566 PR by @jhenkens. + - #1564 issue by @jhenkens. + + +2.6.0 - 2020-07-01 +================== + +### Fixes +- Fix node hooks when `NPM_CONFIG_USERCONFIG` is set + - #1521 PR by @asottile. + - #1516 issue by @rkm. + +### Features +- Skip `rbenv` / `ruby-download` if system ruby is available + - #1509 PR by @asottile. +- Partial support for ruby on windows (if system ruby is installed) + - #1509 PR by @asottile. + - #201 issue by @asottile. + +2.5.1 - 2020-06-09 +================== + +### Fixes +- Prevent infinite recursion of post-checkout on clone + - #1497 PR by @asottile. + - #1496 issue by @admorgan. + +2.5.0 - 2020-06-08 +================== + +### Features +- Expose a `PRE_COMMIT=1` environment variable when running hooks + - #1467 PR by @tech-chad. + - #1426 issue by @lorenzwalthert. + +### Fixes +- Fix `UnicodeDecodeError` on windows when using the `py` launcher to detect + executables with non-ascii characters in the path + - #1474 PR by @asottile. + - #1472 issue by DrFobos. +- Fix `DeprecationWarning` on python3.9 for `random.shuffle` method + - #1480 PR by @asottile. + - #1479 issue by @isidentical. +- Normalize slashes earlier such that global `files` / `exclude` use forward + slashes on windows as well. + - #1494 PR by @asottile. + - #1476 issue by @harrybiddle. + +2.4.0 - 2020-05-11 +================== + +### Features +- Add support for `post-commit` hooks + - #1415 PR by @ModischFabrications. + - #1411 issue by @ModischFabrications. +- Silence pip version warning in python installation error + - #1412 PR by @asottile. +- Improve python `healthy()` when upgrading operating systems. + - #1431 PR by @asottile. + - #1427 issue by @ahonnecke. +- `language: python_venv` is now an alias to `language: python` (and will be + removed in a future version). + - #1431 PR by @asottile. +- Speed up python `healthy()` check. + - #1431 PR by @asottile. +- `pre-commit autoupdate` now tries to maintain quoting style of `rev`. + - #1435 PR by @marcjay. + - #1434 issue by @marcjay. + +### Fixes +- Fix installation of go modules in `repo: local`. + - #1428 PR by @scop. +- Fix committing with unstaged files and a failing `post-checkout` hook. + - #1422 PR by @domodwyer. + - #1418 issue by @domodwyer. +- Fix installation of node hooks with system node installed on freebsd. + - #1443 PR by @asottile. + - #1440 issue by @jockej. +- Fix ruby hooks when `GEM_PATH` is set globally. + - #1442 PR by @tdeo. +- Improve error message when `pre-commit autoupdate` / + `pre-commit migrate-config` are run but the pre-commit configuration is not + valid yaml. + - #1448 PR by @asottile. + - #1447 issue by @rpdelaney. + +2.3.0 - 2020-04-22 +================== + +### Features +- Calculate character width using `east_asian_width` + - #1378 PR by @sophgn. +- Use `language_version: system` by default for `node` hooks if `node` / `npm` + are globally installed. + - #1388 PR by @asottile. + +### Fixes +- No longer use a hard-coded user id for docker hooks on windows + - #1371 PR by @killuazhu. +- Fix colors on windows during `git commit` + - #1381 issue by @Cielquan. + - #1382 PR by @asottile. +- Produce readable error message for incorrect argument count to `hook-impl` + - #1394 issue by @pip9ball. + - #1395 PR by @asottile. +- Fix installations which involve an upgrade of `pip` on windows + - #1398 issue by @xiaohuazi123. + - #1399 PR by @asottile. +- Preserve line endings in `pre-commit autoupdate` + - #1402 PR by @utek. + +2.2.0 - 2020-03-12 +================== + +### Features +- Add support for the `post-checkout` hook + - #1120 issue by @domenkozar. + - #1339 PR by @andrewhare. +- Add more readable `--from-ref` / `--to-ref` aliases for `--source` / + `--origin` + - #1343 PR by @asottile. + +### Fixes +- Make sure that `--commit-msg-filename` is passed for `commit-msg` / + `prepare-commit-msg`. + - #1336 PR by @particledecay. + - #1341 PR by @particledecay. +- Fix crash when installation error is un-decodable bytes + - #1358 issue by @Guts. + - #1359 PR by @asottile. +- Fix python `healthy()` check when `python` executable goes missing. + - #1363 PR by @asottile. +- Fix crash when script executables are missing shebangs. + - #1350 issue by @chriselion. + - #1364 PR by @asottile. + +### Misc. +- pre-commit now requires python>=3.6.1 (previously 3.6.0) + - #1346 PR by @asottile. + +2.1.1 - 2020-02-24 +================== + +### Fixes +- Temporarily restore python 3.6.0 support (broken in 2.0.0) + - reported by @obestwalter. + - 081f3028 by @asottile. + +2.1.0 - 2020-02-18 +================== + +### Features +- Replace `aspy.yaml` with `sort_keys=False`. + - #1306 PR by @asottile. +- Add support for `perl`. + - #1303 PR by @scop. + +### Fixes +- Improve `.git/hooks/*` shebang creation when pythons are in `/usr/local/bin`. + - #1312 issue by @kbsezginel. + - #1319 PR by @asottile. + +### Misc. +- Add repository badge for pre-commit. + - [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit) + - #1334 PR by @ddelange. + +2.0.1 - 2020-01-29 +================== + +### Fixes +- Fix `ImportError` in python 3.6.0 / 3.6.1 for `typing.NoReturn`. + - #1302 PR by @asottile. + +2.0.0 - 2020-01-28 +================== + +### Features +- Expose `PRE_COMMIT_REMOTE_NAME` and `PRE_COMMIT_REMOTE_URL` as environment + variables during `pre-push` hooks. + - #1274 issue by @dmbarreiro. + - #1288 PR by @dmbarreiro. + +### Fixes +- Fix `python -m pre_commit --version` to mention `pre-commit` instead of + `__main__.py`. + - #1273 issue by @ssbarnea. + - #1276 PR by @orcutt989. +- Don't filter `GIT_SSL_NO_VERIFY` from environment when cloning. + - #1293 PR by @schiermike. +- Allow `pre-commit init-templatedir` to succeed even if `core.hooksPath` is + set. + - #1298 issue by @damienrj. + - #1299 PR by @asottile. + +### Misc +- Fix changelog date for 1.21.0. + - #1275 PR by @flaudisio. + +### Updating +- Removed `pcre` language, use `pygrep` instead. + - #1268 PR by @asottile. +- Removed `--tags-only` argument to `pre-commit autoupdate` (it has done + nothing since 0.14.0). + - #1269 by @asottile. +- Remove python2 / python3.5 support. Note that pre-commit still supports + running hooks written in python2, but pre-commit itself requires python 3.6+. + - #1260 issue by @asottile. + - #1277 PR by @asottile. + - #1281 PR by @asottile. + - #1282 PR by @asottile. + - #1287 PR by @asottile. + - #1289 PR by @asottile. + - #1292 PR by @asottile. + +1.21.0 - 2020-01-02 +=================== + +### Features +- Add `conda` as a new `language`. + - #1204 issue by @xhochy. + - #1232 PR by @xhochy. +- Add top-level configuration `files` for file selection. + - #1220 issue by @TheButlah. + - #1248 PR by @asottile. +- Rework `--verbose` / `verbose` to be more consistent with normal runs. + - #1249 PR by @asottile. +- Add support for the `pre-merge-commit` git hook. + - #1210 PR by @asottile. + - this requires git 2.24+. +- Add `pre-commit autoupdate --freeze` which produces "frozen" revisions. + - #1068 issue by @SkypLabs. + - #1256 PR by @asottile. +- Display hook runtime duration when run with `--verbose`. + - #1144 issue by @potiuk. + - #1257 PR by @asottile. + +### Fixes +- Produce better error message when erroneously running inside of `.git`. + - #1219 issue by @Nusserdt. + - #1224 PR by @asottile. + - Note: `git` has since fixed this bug: git/git@36fd304d +- Produce better error message when hook installation fails. + - #1250 issue by @asottile. + - #1251 PR by @asottile. +- Fix cloning when `GIT_SSL_CAINFO` is necessary. + - #1253 issue by @igankevich. + - #1254 PR by @igankevich. +- Fix `pre-commit try-repo` for bare, on-disk repositories. + - #1258 issue by @webknjaz. + - #1259 PR by @asottile. +- Add some whitespace to `pre-commit autoupdate` to improve terminal autolink. + - #1261 issue by @yhoiseth. + - #1262 PR by @yhoiseth. + +### Misc. +- Minor code documentation updates. + - #1200 PR by @ryanrhee. + - #1201 PR by @ryanrhee. + +1.20.0 - 2019-10-28 +=================== + +### Features +- Allow building newer versions of `ruby`. + - #1193 issue by @choffee. + - #1195 PR by @choffee. +- Bump versions reported in `pre-commit sample-config`. + - #1197 PR by @asottile. + +### Fixes +- Fix rare race condition with multiple concurrent first-time runs. + - #1192 issue by @raholler. + - #1196 PR by @asottile. + +1.19.0 - 2019-10-26 +=================== + +### Features +- Allow `--hook-type` to be specified multiple times. + - example: `pre-commit install --hook-type pre-commit --hook-type pre-push` + - #1139 issue by @MaxymVlasov. + - #1145 PR by @asottile. +- Include more version information in crash logs. + - #1142 by @marqueewinq. +- Hook colors are now passed through on platforms which support `pty`. + - #1169 by @asottile. +- pre-commit now uses `importlib.metadata` directly when running in python 3.8 + - #1176 by @asottile. +- Normalize paths to forward slash separators on windows. + - makes it easier to match paths with `files:` regex + - avoids some quoting bugs in shell-based hooks + - #1173 issue by @steigenTI. + - #1179 PR by @asottile. + +### Fixes +- Remove some extra newlines from error messages. + - #1148 by @asottile. +- When a hook is not executable it now reports `not executable` instead of + `not found`. + - #1159 issue by @nixjdm. + - #1161 PR by @WillKoehrsen. +- Fix interleaving of stdout / stderr in hooks. + - #1168 by @asottile. +- Fix python environment `healthy()` check when current working directory + contains modules which shadow standard library names. + - issue by @vwhsu92. + - #1185 PR by @asottile. + +### Updating +- Regexes handling both backslashes and forward slashes for directory + separators now only need to handle forward slashes. + +1.18.3 - 2019-08-27 +=================== + +### Fixes +- Fix `node_modules` plugin installation on windows + - #1123 issue by @henryykt. + - #1122 PR by @henryykt. + +1.18.2 - 2019-08-15 +=================== + +### Fixes +- Make default python lookup more deterministic to avoid redundant installs + - #1117 PR by @scop. + 1.18.1 - 2019-08-11 =================== @@ -251,7 +1474,7 @@ - #881 issue by @henniss. - #912 PR by @asottile. -### Misc +### Misc. - Use `--no-gpg-sign` when running tests - #894 PR by @s0undt3ch. @@ -261,7 +1484,7 @@ ### Features - Run hooks in parallel - - individual hooks may opt out of parallel exection with `require_serial: true` + - individual hooks may opt out of parallel execution with `require_serial: true` - #510 issue by @chriskuehl. - #851 PR by @chriskuehl. @@ -282,7 +1505,7 @@ instead using `--no-document`. - #889 PR by @asottile. -### Misc +### Misc. - Use `--no-gpg-sign` when running tests - #885 PR by @s0undt3ch. @@ -371,7 +1594,7 @@ - #772 issue by @asottile. - #803 PR by @mblayman. -### Misc +### Misc. - Improve travis-ci build times by caching rust / swift artifacts - #781 PR by @expobrain. - Test against python3.7 @@ -440,7 +1663,7 @@ ### Fixes - Fix integration with go 1.10 and `pkg` directory - #725 PR by @asottile -- Restore support for `git<1.8.5` (inadvertantly removed in 1.7.0) +- Restore support for `git<1.8.5` (inadvertently removed in 1.7.0) - #723 issue by @JohnLyman. - #724 PR by @asottile. @@ -480,7 +1703,7 @@ - #590 issue by @coldnight. - #711 PR by @asottile. -### Misc +### Misc. - test against swift 4.x - #709 by @theresama. @@ -524,7 +1747,7 @@ - #200 issue by @asottile. - #685 PR by @asottile. -### Misc +### Misc. - internal reorganization of `PrefixedCommandRunner` -> `Prefix` - #684 PR by @asottile. - https-ify links. @@ -539,7 +1762,7 @@ - Fix `local` golang repositories with `additional_dependencies`. - #679 #680 issue and PR by @asottile. -### Misc +### Misc. - Replace some string literals with constants - #678 PR by @revolter. @@ -747,7 +1970,7 @@ that have helped us get this far! 0.18.1 - 2017-09-04 =================== - Only mention locking when waiting for a lock. -- Fix `IOError` during locking in timeout situtation on windows under python 2. +- Fix `IOError` during locking in timeout situation on windows under python 2. 0.18.0 - 2017-09-02 =================== @@ -869,7 +2092,7 @@ that have helped us get this far! 0.13.1 - 2017-02-16 =================== -- Fix dummy gem for ruby local hooks +- Fix placeholder gem for ruby local hooks 0.13.0 - 2017-02-16 =================== diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2b83c823..da7f9432 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,12 +4,15 @@ - The complete test suite depends on having at least the following installed (possibly not a complete list) - - git (A sufficiently newer version is required to run pre-push tests) - - python2 (Required by a test which checks different python versions) + - git (Version 2.24.0 or above is required to run pre-merge-commit tests) - python3 (Required by a test which checks different python versions) - tox (or virtualenv) - ruby + gem - docker + - conda + - cargo (required by tests for rust dependencies) + - go (required by tests for go dependencies) + - swift ### Setting up an environment @@ -17,7 +20,8 @@ This is useful for running specific tests. The easiest way to set this up is to run: 1. `tox --devenv venv` (note: requires tox>=3.13) -2. `. venv/bin/activate` +2. `. venv/bin/activate` (or follow the [activation instructions] for your + platform) This will create and put you into a virtualenv which has an editable installation of pre-commit. Hack away! Running `pre-commit` will reflect @@ -60,14 +64,14 @@ to implement. The current implemented languages are at varying levels: - 0th class - pre-commit does not require any dependencies for these languages as they're not actually languages (current examples: fail, pygrep) - 1st class - pre-commit will bootstrap a full interpreter requiring nothing to - be installed globally (current examples: node, ruby) + be installed globally (current examples: go, node, ruby, rust) - 2nd class - pre-commit requires the user to install the language globally but - will install tools in an isolated fashion (current examples: python, go, rust, - swift, docker). + will install tools in an isolated fashion (current examples: python, swift, + docker). - 3rd class - pre-commit requires the user to install both the tool and the language globally (current examples: script, system) -"third class" is usually the easiest to implement first and is perfectly +"second class" is usually the easiest to implement first and is perfectly acceptable. Ideally the language works on the supported platforms for pre-commit (linux, @@ -88,7 +92,7 @@ language, for example: here are the apis that should be implemented for a language -Note that these are also documented in [`pre_commit/languages/all.py`](https://github.com/pre-commit/pre-commit/blob/master/pre_commit/languages/all.py) +Note that these are also documented in [`pre_commit/lang_base.py`](https://github.com/pre-commit/pre-commit/blob/main/pre_commit/lang_base.py) #### `ENVIRONMENT_DIR` @@ -107,20 +111,21 @@ one cannot be determined, return `'default'`. You generally don't need to implement this on a first pass and can just use: ```python -get_default_version = helpers.basic_default_version +get_default_version = lang_base.basic_default_version ``` `python` is currently the only language which implements this api -#### `healthy` +#### `health_check` This is used to check whether the installed environment is considered healthy. -This function should return `True` or `False`. +This function should return a detailed message if unhealthy or `None` if +healthy. You generally don't need to implement this on a first pass and can just use: ```python -healthy = helpers.basic_healthy +health_check = lang_base.basic_health_check ``` `python` is currently the only language which implements this api, for python @@ -132,7 +137,7 @@ this is the trickiest one to implement and where all the smart parts happen. this api should do the following things -- (0th / 3rd class): `install_environment = helpers.no_install` +- (0th / 3rd class): `install_environment = lang_base.no_install` - (1st class): install a language runtime into the hook's directory - (2nd class): install the package at `.` into the `ENVIRONMENT_DIR` - (2nd class, optional): install packages listed in `additional_dependencies` @@ -144,3 +149,5 @@ This is usually the easiest to implement, most of them look the same as the `node` hook implementation: https://github.com/pre-commit/pre-commit/blob/160238220f022035c8ef869c9a8642f622c02118/pre_commit/languages/node.py#L72-L74 + +[activation instructions]: https://virtualenv.pypa.io/en/latest/user_guide.html#activators diff --git a/README.md b/README.md index 01d0d757..0c81a789 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -[![Build Status](https://dev.azure.com/asottile/asottile/_apis/build/status/pre-commit.pre-commit?branchName=master)](https://dev.azure.com/asottile/asottile/_build/latest?definitionId=21&branchName=master) -[![Azure DevOps coverage](https://img.shields.io/azure-devops/coverage/asottile/asottile/21/master.svg)](https://dev.azure.com/asottile/asottile/_build/latest?definitionId=21&branchName=master) +[![build status](https://github.com/pre-commit/pre-commit/actions/workflows/main.yml/badge.svg)](https://github.com/pre-commit/pre-commit/actions/workflows/main.yml) +[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/pre-commit/pre-commit/main.svg)](https://results.pre-commit.ci/latest/github/pre-commit/pre-commit/main) ## pre-commit diff --git a/azure-pipelines.yml b/azure-pipelines.yml deleted file mode 100644 index 30b873a0..00000000 --- a/azure-pipelines.yml +++ /dev/null @@ -1,46 +0,0 @@ -trigger: - branches: - include: [master, test-me-*] - tags: - include: ['*'] - -resources: - repositories: - - repository: asottile - type: github - endpoint: github - name: asottile/azure-pipeline-templates - ref: refs/tags/v0.0.15 - -jobs: -- template: job--pre-commit.yml@asottile -- template: job--python-tox.yml@asottile - parameters: - toxenvs: [py27, py37] - os: windows - additional_variables: - COVERAGE_IGNORE_WINDOWS: '# pragma: windows no cover' - TOX_TESTENV_PASSENV: COVERAGE_IGNORE_WINDOWS - TEMP: C:\Temp # remove when dropping python2 -- template: job--python-tox.yml@asottile - parameters: - toxenvs: [py37] - os: linux - name_postfix: _latest_git - pre_test: - - task: UseRubyVersion@0 - - template: step--git-install.yml - - bash: | - testing/get-swift.sh - echo '##vso[task.prependpath]/tmp/swift/usr/bin' - displayName: install swift -- template: job--python-tox.yml@asottile - parameters: - toxenvs: [pypy, pypy3, py27, py36, py37] - os: linux - pre_test: - - task: UseRubyVersion@0 - - bash: | - testing/get-swift.sh - echo '##vso[task.prependpath]/tmp/swift/usr/bin' - displayName: install swift diff --git a/pre_commit/__main__.py b/pre_commit/__main__.py index fc424d82..bda61eec 100644 --- a/pre_commit/__main__.py +++ b/pre_commit/__main__.py @@ -1,7 +1,7 @@ -from __future__ import absolute_import +from __future__ import annotations from pre_commit.main import main if __name__ == '__main__': - exit(main()) + raise SystemExit(main()) diff --git a/pre_commit/all_languages.py b/pre_commit/all_languages.py new file mode 100644 index 00000000..166bc167 --- /dev/null +++ b/pre_commit/all_languages.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +from pre_commit.lang_base import Language +from pre_commit.languages import conda +from pre_commit.languages import coursier +from pre_commit.languages import dart +from pre_commit.languages import docker +from pre_commit.languages import docker_image +from pre_commit.languages import dotnet +from pre_commit.languages import fail +from pre_commit.languages import golang +from pre_commit.languages import haskell +from pre_commit.languages import julia +from pre_commit.languages import lua +from pre_commit.languages import node +from pre_commit.languages import perl +from pre_commit.languages import pygrep +from pre_commit.languages import python +from pre_commit.languages import r +from pre_commit.languages import ruby +from pre_commit.languages import rust +from pre_commit.languages import swift +from pre_commit.languages import unsupported +from pre_commit.languages import unsupported_script + + +languages: dict[str, Language] = { + 'conda': conda, + 'coursier': coursier, + 'dart': dart, + 'docker': docker, + 'docker_image': docker_image, + 'dotnet': dotnet, + 'fail': fail, + 'golang': golang, + 'haskell': haskell, + 'julia': julia, + 'lua': lua, + 'node': node, + 'perl': perl, + 'pygrep': pygrep, + 'python': python, + 'r': r, + 'ruby': ruby, + 'rust': rust, + 'swift': swift, + 'unsupported': unsupported, + 'unsupported_script': unsupported_script, +} +language_names = sorted(languages) diff --git a/pre_commit/clientlib.py b/pre_commit/clientlib.py index 14a22b99..51f14d26 100644 --- a/pre_commit/clientlib.py +++ b/pre_commit/clientlib.py @@ -1,65 +1,252 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import argparse import functools import logging -import pipes +import os.path +import re +import shlex import sys +from collections.abc import Callable +from collections.abc import Sequence +from typing import Any +from typing import NamedTuple import cfgv -from aspy.yaml import ordered_load from identify.identify import ALL_TAGS import pre_commit.constants as C -from pre_commit.error_handler import FatalError -from pre_commit.languages.all import all_languages -from pre_commit.util import parse_version +from pre_commit.all_languages import language_names +from pre_commit.errors import FatalError +from pre_commit.yaml import yaml_load logger = logging.getLogger('pre_commit') +check_string_regex = cfgv.check_and(cfgv.check_string, cfgv.check_regex) -def check_type_tag(tag): +HOOK_TYPES = ( + 'commit-msg', + 'post-checkout', + 'post-commit', + 'post-merge', + 'post-rewrite', + 'pre-commit', + 'pre-merge-commit', + 'pre-push', + 'pre-rebase', + 'prepare-commit-msg', +) +# `manual` is not invoked by any installed git hook. See #719 +STAGES = (*HOOK_TYPES, 'manual') + + +def check_type_tag(tag: str) -> None: if tag not in ALL_TAGS: raise cfgv.ValidationError( - 'Type tag {!r} is not recognized. ' - 'Try upgrading identify and pre-commit?'.format(tag), + f'Type tag {tag!r} is not recognized. ' + f'Try upgrading identify and pre-commit?', ) -def check_min_version(version): +def parse_version(s: str) -> tuple[int, ...]: + """poor man's version comparison""" + return tuple(int(p) for p in s.split('.')) + + +def check_min_version(version: str) -> None: if parse_version(version) > parse_version(C.VERSION): raise cfgv.ValidationError( - 'pre-commit version {} is required but version {} is installed. ' - 'Perhaps run `pip install --upgrade pre-commit`.'.format( - version, C.VERSION, - ), + f'pre-commit version {version} is required but version ' + f'{C.VERSION} is installed. ' + f'Perhaps run `pip install --upgrade pre-commit`.', ) -def _make_argparser(filenames_help): - parser = argparse.ArgumentParser() - parser.add_argument('filenames', nargs='*', help=filenames_help) - parser.add_argument('-V', '--version', action='version', version=C.VERSION) - return parser +_STAGES = { + 'commit': 'pre-commit', + 'merge-commit': 'pre-merge-commit', + 'push': 'pre-push', +} + + +def transform_stage(stage: str) -> str: + return _STAGES.get(stage, stage) + + +MINIMAL_MANIFEST_SCHEMA = cfgv.Array( + cfgv.Map( + 'Hook', 'id', + cfgv.Required('id', cfgv.check_string), + cfgv.Optional('stages', cfgv.check_array(cfgv.check_string), []), + ), +) + + +def warn_for_stages_on_repo_init(repo: str, directory: str) -> None: + try: + manifest = cfgv.load_from_filename( + os.path.join(directory, C.MANIFEST_FILE), + schema=MINIMAL_MANIFEST_SCHEMA, + load_strategy=yaml_load, + exc_tp=InvalidManifestError, + ) + except InvalidManifestError: + return # they'll get a better error message when it actually loads! + + legacy_stages = {} # sorted set + for hook in manifest: + for stage in hook.get('stages', ()): + if stage in _STAGES: + legacy_stages[stage] = True + + if legacy_stages: + logger.warning( + f'repo `{repo}` uses deprecated stage names ' + f'({", ".join(legacy_stages)}) which will be removed in a ' + f'future version. ' + f'Hint: often `pre-commit autoupdate --repo {shlex.quote(repo)}` ' + f'will fix this. ' + f'if it does not -- consider reporting an issue to that repo.', + ) + + +class StagesMigrationNoDefault(NamedTuple): + key: str + default: Sequence[str] + + def check(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + return + + with cfgv.validate_context(f'At key: {self.key}'): + val = dct[self.key] + cfgv.check_array(cfgv.check_any)(val) + + val = [transform_stage(v) for v in val] + cfgv.check_array(cfgv.check_one_of(STAGES))(val) + + def apply_default(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + return + dct[self.key] = [transform_stage(v) for v in dct[self.key]] + + def remove_default(self, dct: dict[str, Any]) -> None: + raise NotImplementedError + + +class StagesMigration(StagesMigrationNoDefault): + def apply_default(self, dct: dict[str, Any]) -> None: + dct.setdefault(self.key, self.default) + super().apply_default(dct) + + +class DeprecatedStagesWarning(NamedTuple): + key: str + + def check(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + return + + val = dct[self.key] + cfgv.check_array(cfgv.check_any)(val) + + legacy_stages = [stage for stage in val if stage in _STAGES] + if legacy_stages: + logger.warning( + f'hook id `{dct["id"]}` uses deprecated stage names ' + f'({", ".join(legacy_stages)}) which will be removed in a ' + f'future version. ' + f'run: `pre-commit migrate-config` to automatically fix this.', + ) + + def apply_default(self, dct: dict[str, Any]) -> None: + pass + + def remove_default(self, dct: dict[str, Any]) -> None: + raise NotImplementedError + + +class DeprecatedDefaultStagesWarning(NamedTuple): + key: str + + def check(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + return + + val = dct[self.key] + cfgv.check_array(cfgv.check_any)(val) + + legacy_stages = [stage for stage in val if stage in _STAGES] + if legacy_stages: + logger.warning( + f'top-level `default_stages` uses deprecated stage names ' + f'({", ".join(legacy_stages)}) which will be removed in a ' + f'future version. ' + f'run: `pre-commit migrate-config` to automatically fix this.', + ) + + def apply_default(self, dct: dict[str, Any]) -> None: + pass + + def remove_default(self, dct: dict[str, Any]) -> None: + raise NotImplementedError + + +def _translate_language(name: str) -> str: + return { + 'system': 'unsupported', + 'script': 'unsupported_script', + }.get(name, name) + + +class LanguageMigration(NamedTuple): # remove + key: str + check_fn: Callable[[object], None] + + def check(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + return + + with cfgv.validate_context(f'At key: {self.key}'): + self.check_fn(_translate_language(dct[self.key])) + + def apply_default(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + return + + dct[self.key] = _translate_language(dct[self.key]) + + def remove_default(self, dct: dict[str, Any]) -> None: + raise NotImplementedError + + +class LanguageMigrationRequired(LanguageMigration): # replace with Required + def check(self, dct: dict[str, Any]) -> None: + if self.key not in dct: + raise cfgv.ValidationError(f'Missing required key: {self.key}') + + super().check(dct) MANIFEST_HOOK_DICT = cfgv.Map( 'Hook', 'id', + # check first in case it uses some newer, incompatible feature + cfgv.Optional( + 'minimum_pre_commit_version', + cfgv.check_and(cfgv.check_string, check_min_version), + '0', + ), + cfgv.Required('id', cfgv.check_string), cfgv.Required('name', cfgv.check_string), cfgv.Required('entry', cfgv.check_string), - cfgv.Required('language', cfgv.check_one_of(all_languages)), + LanguageMigrationRequired('language', cfgv.check_one_of(language_names)), cfgv.Optional('alias', cfgv.check_string, ''), - cfgv.Optional( - 'files', cfgv.check_and(cfgv.check_string, cfgv.check_regex), '', - ), - cfgv.Optional( - 'exclude', cfgv.check_and(cfgv.check_string, cfgv.check_regex), '^$', - ), + cfgv.Optional('files', check_string_regex, ''), + cfgv.Optional('exclude', check_string_regex, '^$'), cfgv.Optional('types', cfgv.check_array(check_type_tag), ['file']), + cfgv.Optional('types_or', cfgv.check_array(check_type_tag), []), cfgv.Optional('exclude_types', cfgv.check_array(check_type_tag), []), cfgv.Optional( @@ -67,13 +254,13 @@ MANIFEST_HOOK_DICT = cfgv.Map( ), cfgv.Optional('args', cfgv.check_array(cfgv.check_string), []), cfgv.Optional('always_run', cfgv.check_bool, False), + cfgv.Optional('fail_fast', cfgv.check_bool, False), cfgv.Optional('pass_filenames', cfgv.check_bool, True), cfgv.Optional('description', cfgv.check_string, ''), cfgv.Optional('language_version', cfgv.check_string, C.DEFAULT), cfgv.Optional('log_file', cfgv.check_string, ''), - cfgv.Optional('minimum_pre_commit_version', cfgv.check_string, '0'), cfgv.Optional('require_serial', cfgv.check_bool, False), - cfgv.Optional('stages', cfgv.check_array(cfgv.check_one_of(C.STAGES)), []), + StagesMigration('stages', []), cfgv.Optional('verbose', cfgv.check_bool, False), ) MANIFEST_SCHEMA = cfgv.Array(MANIFEST_HOOK_DICT) @@ -83,83 +270,106 @@ class InvalidManifestError(FatalError): pass +def _load_manifest_forward_compat(contents: str) -> object: + obj = yaml_load(contents) + if isinstance(obj, dict): + check_min_version('5') + raise AssertionError('unreachable') + else: + return obj + + load_manifest = functools.partial( cfgv.load_from_filename, schema=MANIFEST_SCHEMA, - load_strategy=ordered_load, + load_strategy=_load_manifest_forward_compat, exc_tp=InvalidManifestError, ) -def validate_manifest_main(argv=None): - parser = _make_argparser('Manifest filenames.') - args = parser.parse_args(argv) - ret = 0 - for filename in args.filenames: - try: - load_manifest(filename) - except InvalidManifestError as e: - print(e) - ret = 1 - return ret - - LOCAL = 'local' META = 'meta' -class MigrateShaToRev(object): - key = 'rev' +class WarnMutableRev(cfgv.Conditional): + def check(self, dct: dict[str, Any]) -> None: + super().check(dct) - @staticmethod - def _cond(key): - return cfgv.Conditional( - key, cfgv.check_string, - condition_key='repo', - condition_value=cfgv.NotIn(LOCAL, META), - ensure_absent=True, - ) + if self.key in dct: + rev = dct[self.key] - def check(self, dct): - if dct.get('repo') in {LOCAL, META}: - self._cond('rev').check(dct) - self._cond('sha').check(dct) - elif 'sha' in dct and 'rev' in dct: - raise cfgv.ValidationError('Cannot specify both sha and rev') - elif 'sha' in dct: - self._cond('sha').check(dct) - else: - self._cond('rev').check(dct) - - def apply_default(self, dct): - if 'sha' in dct: - dct['rev'] = dct.pop('sha') - - def remove_default(self, dct): - pass + if '.' not in rev and not re.match(r'^[a-fA-F0-9]+$', rev): + logger.warning( + f'The {self.key!r} field of repo {dct["repo"]!r} ' + f'appears to be a mutable reference ' + f'(moving tag / branch). Mutable references are never ' + f'updated after first install and are not supported. ' + f'See https://pre-commit.com/#using-the-latest-version-for-a-repository ' # noqa: E501 + f'for more details. ' + f'Hint: `pre-commit autoupdate` often fixes this.', + ) -def _entry(modname): +class OptionalSensibleRegexAtHook(cfgv.OptionalNoDefault): + def check(self, dct: dict[str, Any]) -> None: + super().check(dct) + + if '/*' in dct.get(self.key, ''): + logger.warning( + f'The {self.key!r} field in hook {dct.get("id")!r} is a ' + f"regex, not a glob -- matching '/*' probably isn't what you " + f'want here', + ) + for fwd_slash_re in (r'[\\/]', r'[\/]', r'[/\\]'): + if fwd_slash_re in dct.get(self.key, ''): + logger.warning( + fr'pre-commit normalizes slashes in the {self.key!r} ' + fr'field in hook {dct.get("id")!r} to forward slashes, ' + fr'so you can use / instead of {fwd_slash_re}', + ) + + +class OptionalSensibleRegexAtTop(cfgv.OptionalNoDefault): + def check(self, dct: dict[str, Any]) -> None: + super().check(dct) + + if '/*' in dct.get(self.key, ''): + logger.warning( + f'The top-level {self.key!r} field is a regex, not a glob -- ' + f"matching '/*' probably isn't what you want here", + ) + for fwd_slash_re in (r'[\\/]', r'[\/]', r'[/\\]'): + if fwd_slash_re in dct.get(self.key, ''): + logger.warning( + fr'pre-commit normalizes the slashes in the top-level ' + fr'{self.key!r} field to forward slashes, so you ' + fr'can use / instead of {fwd_slash_re}', + ) + + +def _entry(modname: str) -> str: """the hook `entry` is passed through `shlex.split()` by the command runner, so to prevent issues with spaces and backslashes (on Windows) it must be quoted here. """ - return '{} -m pre_commit.meta_hooks.{}'.format( - pipes.quote(sys.executable), modname, - ) + return f'{shlex.quote(sys.executable)} -m pre_commit.meta_hooks.{modname}' -def warn_unknown_keys_root(extra, orig_keys, dct): +def warn_unknown_keys_root( + extra: Sequence[str], + orig_keys: Sequence[str], + dct: dict[str, str], +) -> None: + logger.warning(f'Unexpected key(s) present at root: {", ".join(extra)}') + + +def warn_unknown_keys_repo( + extra: Sequence[str], + orig_keys: Sequence[str], + dct: dict[str, str], +) -> None: logger.warning( - 'Unexpected key(s) present at root: {}'.format(', '.join(extra)), - ) - - -def warn_unknown_keys_repo(extra, orig_keys, dct): - logger.warning( - 'Unexpected key(s) present on {}: {}'.format( - dct['repo'], ', '.join(extra), - ), + f'Unexpected key(s) present on {dct["repo"]}: {", ".join(extra)}', ) @@ -167,14 +377,14 @@ _meta = ( ( 'check-hooks-apply', ( ('name', 'Check hooks apply to the repository'), - ('files', C.CONFIG_FILE), + ('files', f'^{re.escape(C.CONFIG_FILE)}$'), ('entry', _entry('check_hooks_apply')), ), ), ( 'check-useless-excludes', ( ('name', 'Check for useless excludes'), - ('files', C.CONFIG_FILE), + ('files', f'^{re.escape(C.CONFIG_FILE)}$'), ('entry', _entry('check_useless_excludes')), ), ), @@ -187,25 +397,44 @@ _meta = ( ), ) + +class NotAllowed(cfgv.OptionalNoDefault): + def check(self, dct: dict[str, Any]) -> None: + if self.key in dct: + raise cfgv.ValidationError(f'{self.key!r} cannot be overridden') + + +_COMMON_HOOK_WARNINGS = ( + OptionalSensibleRegexAtHook('files', cfgv.check_string), + OptionalSensibleRegexAtHook('exclude', cfgv.check_string), + DeprecatedStagesWarning('stages'), +) + META_HOOK_DICT = cfgv.Map( 'Hook', 'id', cfgv.Required('id', cfgv.check_string), cfgv.Required('id', cfgv.check_one_of(tuple(k for k, _ in _meta))), - # language must be system - cfgv.Optional('language', cfgv.check_one_of({'system'}), 'system'), - *([ + # language must be `unsupported` + cfgv.Optional( + 'language', cfgv.check_one_of({'unsupported'}), 'unsupported', + ), + # entry cannot be overridden + NotAllowed('entry', cfgv.check_any), + *( # default to the hook definition for the meta hooks cfgv.ConditionalOptional(key, cfgv.check_any, value, 'id', hook_id) for hook_id, values in _meta for key, value in values - ] + [ + ), + *( # default to the "manifest" parsing cfgv.OptionalNoDefault(item.key, item.check_fn) # these will always be defaulted above if item.key in {'name', 'language', 'entry'} else item for item in MANIFEST_HOOK_DICT.items - ]) + ), + *_COMMON_HOOK_WARNINGS, ) CONFIG_HOOK_DICT = cfgv.Map( 'Hook', 'id', @@ -216,11 +445,22 @@ CONFIG_HOOK_DICT = cfgv.Map( # are optional. # No defaults are provided here as the config is merged on top of the # manifest. - *[ + *( cfgv.OptionalNoDefault(item.key, item.check_fn) for item in MANIFEST_HOOK_DICT.items if item.key != 'id' - ] + if item.key != 'stages' + if item.key != 'language' # remove + ), + StagesMigrationNoDefault('stages', []), + LanguageMigration('language', cfgv.check_one_of(language_names)), # remove + *_COMMON_HOOK_WARNINGS, +) +LOCAL_HOOK_DICT = cfgv.Map( + 'Hook', 'id', + + *MANIFEST_HOOK_DICT.items, + *_COMMON_HOOK_WARNINGS, ) CONFIG_REPO_DICT = cfgv.Map( 'Repository', 'repo', @@ -232,7 +472,7 @@ CONFIG_REPO_DICT = cfgv.Map( 'repo', cfgv.NotIn(LOCAL, META), ), cfgv.ConditionalRecurse( - 'hooks', cfgv.Array(MANIFEST_HOOK_DICT), + 'hooks', cfgv.Array(LOCAL_HOOK_DICT), 'repo', LOCAL, ), cfgv.ConditionalRecurse( @@ -240,44 +480,62 @@ CONFIG_REPO_DICT = cfgv.Map( 'repo', META, ), - MigrateShaToRev(), + WarnMutableRev( + 'rev', cfgv.check_string, + condition_key='repo', + condition_value=cfgv.NotIn(LOCAL, META), + ensure_absent=True, + ), cfgv.WarnAdditionalKeys(('repo', 'rev', 'hooks'), warn_unknown_keys_repo), ) DEFAULT_LANGUAGE_VERSION = cfgv.Map( 'DefaultLanguageVersion', None, - cfgv.NoAdditionalKeys(all_languages), - *[cfgv.Optional(x, cfgv.check_string, C.DEFAULT) for x in all_languages] + cfgv.NoAdditionalKeys(language_names), + *(cfgv.Optional(x, cfgv.check_string, C.DEFAULT) for x in language_names), ) CONFIG_SCHEMA = cfgv.Map( 'Config', None, - cfgv.RequiredRecurse('repos', cfgv.Array(CONFIG_REPO_DICT)), - cfgv.OptionalRecurse( - 'default_language_version', DEFAULT_LANGUAGE_VERSION, {}, - ), - cfgv.Optional( - 'default_stages', - cfgv.check_array(cfgv.check_one_of(C.STAGES)), - C.STAGES, - ), - cfgv.Optional('exclude', cfgv.check_regex, '^$'), - cfgv.Optional('fail_fast', cfgv.check_bool, False), + # check first in case it uses some newer, incompatible feature cfgv.Optional( 'minimum_pre_commit_version', cfgv.check_and(cfgv.check_string, check_min_version), '0', ), + + cfgv.RequiredRecurse('repos', cfgv.Array(CONFIG_REPO_DICT)), + cfgv.Optional( + 'default_install_hook_types', + cfgv.check_array(cfgv.check_one_of(HOOK_TYPES)), + ['pre-commit'], + ), + cfgv.OptionalRecurse( + 'default_language_version', DEFAULT_LANGUAGE_VERSION, {}, + ), + StagesMigration('default_stages', STAGES), + DeprecatedDefaultStagesWarning('default_stages'), + cfgv.Optional('files', check_string_regex, ''), + cfgv.Optional('exclude', check_string_regex, '^$'), + cfgv.Optional('fail_fast', cfgv.check_bool, False), cfgv.WarnAdditionalKeys( ( 'repos', + 'default_install_hook_types', 'default_language_version', 'default_stages', + 'files', 'exclude', 'fail_fast', 'minimum_pre_commit_version', + 'ci', ), warn_unknown_keys_root, ), + OptionalSensibleRegexAtTop('files', cfgv.check_string), + OptionalSensibleRegexAtTop('exclude', cfgv.check_string), + + # do not warn about configuration for pre-commit.ci + cfgv.OptionalNoDefault('ci', cfgv.check_type(dict)), ) @@ -285,31 +543,9 @@ class InvalidConfigError(FatalError): pass -def ordered_load_normalize_legacy_config(contents): - data = ordered_load(contents) - if isinstance(data, list): - # TODO: Once happy, issue a deprecation warning and instructions - return {'repos': data} - else: - return data - - load_config = functools.partial( cfgv.load_from_filename, schema=CONFIG_SCHEMA, - load_strategy=ordered_load_normalize_legacy_config, + load_strategy=yaml_load, exc_tp=InvalidConfigError, ) - - -def validate_config_main(argv=None): - parser = _make_argparser('Config filenames.') - args = parser.parse_args(argv) - ret = 0 - for filename in args.filenames: - try: - load_config(filename) - except InvalidConfigError as e: - print(e) - ret = 1 - return ret diff --git a/pre_commit/color.py b/pre_commit/color.py index 1fb6acce..2d6f248b 100644 --- a/pre_commit/color.py +++ b/pre_commit/color.py @@ -1,28 +1,70 @@ -from __future__ import unicode_literals +from __future__ import annotations +import argparse import os import sys -terminal_supports_color = True -if os.name == 'nt': # pragma: no cover (windows) - from pre_commit.color_windows import enable_virtual_terminal_processing +if sys.platform == 'win32': # pragma: no cover (windows) + def _enable() -> None: + from ctypes import POINTER + from ctypes import windll + from ctypes import WinError + from ctypes import WINFUNCTYPE + from ctypes.wintypes import BOOL + from ctypes.wintypes import DWORD + from ctypes.wintypes import HANDLE + + STD_ERROR_HANDLE = -12 + ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4 + + def bool_errcheck(result, func, args): + if not result: + raise WinError() + return args + + GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)( + ('GetStdHandle', windll.kernel32), ((1, 'nStdHandle'),), + ) + + GetConsoleMode = WINFUNCTYPE(BOOL, HANDLE, POINTER(DWORD))( + ('GetConsoleMode', windll.kernel32), + ((1, 'hConsoleHandle'), (2, 'lpMode')), + ) + GetConsoleMode.errcheck = bool_errcheck + + SetConsoleMode = WINFUNCTYPE(BOOL, HANDLE, DWORD)( + ('SetConsoleMode', windll.kernel32), + ((1, 'hConsoleHandle'), (1, 'dwMode')), + ) + SetConsoleMode.errcheck = bool_errcheck + + # As of Windows 10, the Windows console supports (some) ANSI escape + # sequences, but it needs to be enabled using `SetConsoleMode` first. + # + # More info on the escape sequences supported: + # https://msdn.microsoft.com/en-us/library/windows/desktop/mt638032(v=vs.85).aspx + stderr = GetStdHandle(STD_ERROR_HANDLE) + flags = GetConsoleMode(stderr) + SetConsoleMode(stderr, flags | ENABLE_VIRTUAL_TERMINAL_PROCESSING) + try: - enable_virtual_terminal_processing() - except WindowsError: + _enable() + except OSError: terminal_supports_color = False + else: + terminal_supports_color = True +else: # pragma: win32 no cover + terminal_supports_color = True RED = '\033[41m' GREEN = '\033[42m' YELLOW = '\033[43;30m' TURQUOISE = '\033[46;30m' -NORMAL = '\033[0m' +SUBTLE = '\033[2m' +NORMAL = '\033[m' -class InvalidColorSetting(ValueError): - pass - - -def format_color(text, color, use_color_setting): +def format_color(text: str, color: str, use_color_setting: bool) -> str: """Format text with color. Args: @@ -30,29 +72,38 @@ def format_color(text, color, use_color_setting): color - The color start string use_color_setting - Whether or not to color """ - if not use_color_setting: - return text + if use_color_setting: + return f'{color}{text}{NORMAL}' else: - return '{}{}{}'.format(color, text, NORMAL) + return text COLOR_CHOICES = ('auto', 'always', 'never') -def use_color(setting): +def use_color(setting: str) -> bool: """Choose whether to use color based on the command argument. Args: setting - Either `auto`, `always`, or `never` """ if setting not in COLOR_CHOICES: - raise InvalidColorSetting(setting) + raise ValueError(setting) return ( setting == 'always' or ( setting == 'auto' and - sys.stdout.isatty() and + sys.stderr.isatty() and terminal_supports_color and os.getenv('TERM') != 'dumb' ) ) + + +def add_color_option(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + '--color', default=os.environ.get('PRE_COMMIT_COLOR', 'auto'), + type=use_color, + metavar='{' + ','.join(COLOR_CHOICES) + '}', + help='Whether to use color in output. Defaults to `%(default)s`.', + ) diff --git a/pre_commit/color_windows.py b/pre_commit/color_windows.py deleted file mode 100644 index 9b8555e8..00000000 --- a/pre_commit/color_windows.py +++ /dev/null @@ -1,48 +0,0 @@ -from __future__ import absolute_import -from __future__ import unicode_literals - -from ctypes import POINTER -from ctypes import windll -from ctypes import WinError -from ctypes import WINFUNCTYPE -from ctypes.wintypes import BOOL -from ctypes.wintypes import DWORD -from ctypes.wintypes import HANDLE - -STD_OUTPUT_HANDLE = -11 -ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4 - - -def bool_errcheck(result, func, args): - if not result: - raise WinError() - return args - - -GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)( - ('GetStdHandle', windll.kernel32), ((1, 'nStdHandle'),), -) - -GetConsoleMode = WINFUNCTYPE(BOOL, HANDLE, POINTER(DWORD))( - ('GetConsoleMode', windll.kernel32), - ((1, 'hConsoleHandle'), (2, 'lpMode')), -) -GetConsoleMode.errcheck = bool_errcheck - -SetConsoleMode = WINFUNCTYPE(BOOL, HANDLE, DWORD)( - ('SetConsoleMode', windll.kernel32), - ((1, 'hConsoleHandle'), (1, 'dwMode')), -) -SetConsoleMode.errcheck = bool_errcheck - - -def enable_virtual_terminal_processing(): - """As of Windows 10, the Windows console supports (some) ANSI escape - sequences, but it needs to be enabled using `SetConsoleMode` first. - - More info on the escape sequences supported: - https://msdn.microsoft.com/en-us/library/windows/desktop/mt638032(v=vs.85).aspx - """ - stdout = GetStdHandle(STD_OUTPUT_HANDLE) - flags = GetConsoleMode(stdout) - SetConsoleMode(stdout, flags | ENABLE_VIRTUAL_TERMINAL_PROCESSING) diff --git a/pre_commit/commands/autoupdate.py b/pre_commit/commands/autoupdate.py index fdada185..aa0c5e25 100644 --- a/pre_commit/commands/autoupdate.py +++ b/pre_commit/commands/autoupdate.py @@ -1,18 +1,17 @@ -from __future__ import print_function -from __future__ import unicode_literals +from __future__ import annotations +import concurrent.futures import os.path import re - -import six -from aspy.yaml import ordered_dump -from aspy.yaml import ordered_load -from cfgv import remove_defaults +import tempfile +from collections.abc import Sequence +from typing import Any +from typing import NamedTuple import pre_commit.constants as C from pre_commit import git from pre_commit import output -from pre_commit.clientlib import CONFIG_SCHEMA +from pre_commit import xargs from pre_commit.clientlib import InvalidManifestError from pre_commit.clientlib import load_config from pre_commit.clientlib import load_manifest @@ -21,143 +20,196 @@ from pre_commit.clientlib import META from pre_commit.commands.migrate_config import migrate_config from pre_commit.util import CalledProcessError from pre_commit.util import cmd_output -from pre_commit.util import tmpdir +from pre_commit.util import cmd_output_b +from pre_commit.yaml import yaml_dump +from pre_commit.yaml import yaml_load + + +class RevInfo(NamedTuple): + repo: str + rev: str + frozen: str | None = None + hook_ids: frozenset[str] = frozenset() + + @classmethod + def from_config(cls, config: dict[str, Any]) -> RevInfo: + return cls(config['repo'], config['rev']) + + def update(self, tags_only: bool, freeze: bool) -> RevInfo: + with tempfile.TemporaryDirectory() as tmp: + _git = ('git', *git.NO_FS_MONITOR, '-C', tmp) + + if tags_only: + tag_opt = '--abbrev=0' + else: + tag_opt = '--exact' + tag_cmd = (*_git, 'describe', 'FETCH_HEAD', '--tags', tag_opt) + + git.init_repo(tmp, self.repo) + cmd_output_b(*_git, 'config', 'extensions.partialClone', 'true') + cmd_output_b( + *_git, 'fetch', 'origin', 'HEAD', + '--quiet', '--filter=blob:none', '--tags', + ) + + try: + rev = cmd_output(*tag_cmd)[1].strip() + except CalledProcessError: + rev = cmd_output(*_git, 'rev-parse', 'FETCH_HEAD')[1].strip() + else: + if tags_only: + rev = git.get_best_candidate_tag(rev, tmp) + + frozen = None + if freeze: + exact = cmd_output(*_git, 'rev-parse', rev)[1].strip() + if exact != rev: + rev, frozen = exact, rev + + try: + # workaround for windows -- see #2865 + cmd_output_b(*_git, 'show', f'{rev}:{C.MANIFEST_FILE}') + cmd_output(*_git, 'checkout', rev, '--', C.MANIFEST_FILE) + except CalledProcessError: + pass # this will be caught by manifest validating code + try: + manifest = load_manifest(os.path.join(tmp, C.MANIFEST_FILE)) + except InvalidManifestError as e: + raise RepositoryCannotBeUpdatedError(f'[{self.repo}] {e}') + else: + hook_ids = frozenset(hook['id'] for hook in manifest) + + return self._replace(rev=rev, frozen=frozen, hook_ids=hook_ids) class RepositoryCannotBeUpdatedError(RuntimeError): pass -def _update_repo(repo_config, store, tags_only): - """Updates a repository to the tip of `master`. If the repository cannot - be updated because a hook that is configured does not exist in `master`, - this raises a RepositoryCannotBeUpdatedError - - Args: - repo_config - A config for a repository - """ - with tmpdir() as repo_path: - git.init_repo(repo_path, repo_config['repo']) - cmd_output('git', 'fetch', 'origin', 'HEAD', '--tags', cwd=repo_path) - - tag_cmd = ('git', 'describe', 'FETCH_HEAD', '--tags') - if tags_only: - tag_cmd += ('--abbrev=0',) - else: - tag_cmd += ('--exact',) - try: - rev = cmd_output(*tag_cmd, cwd=repo_path)[1].strip() - except CalledProcessError: - tag_cmd = ('git', 'rev-parse', 'FETCH_HEAD') - rev = cmd_output(*tag_cmd, cwd=repo_path)[1].strip() - - # Don't bother trying to update if our rev is the same - if rev == repo_config['rev']: - return repo_config - - try: - path = store.clone(repo_config['repo'], rev) - manifest = load_manifest(os.path.join(path, C.MANIFEST_FILE)) - except InvalidManifestError as e: - raise RepositoryCannotBeUpdatedError(six.text_type(e)) - +def _check_hooks_still_exist_at_rev( + repo_config: dict[str, Any], + info: RevInfo, +) -> None: # See if any of our hooks were deleted with the new commits hooks = {hook['id'] for hook in repo_config['hooks']} - hooks_missing = hooks - {hook['id'] for hook in manifest} + hooks_missing = hooks - info.hook_ids if hooks_missing: raise RepositoryCannotBeUpdatedError( - 'Cannot update because the tip of master is missing these hooks:\n' - '{}'.format(', '.join(sorted(hooks_missing))), + f'[{info.repo}] Cannot update because the update target is ' + f'missing these hooks: {", ".join(sorted(hooks_missing))}', ) - # Construct a new config with the head rev - new_config = repo_config.copy() - new_config['rev'] = rev - return new_config + +def _update_one( + i: int, + repo: dict[str, Any], + *, + tags_only: bool, + freeze: bool, +) -> tuple[int, RevInfo, RevInfo]: + old = RevInfo.from_config(repo) + new = old.update(tags_only=tags_only, freeze=freeze) + _check_hooks_still_exist_at_rev(repo, new) + return i, old, new -REV_LINE_RE = re.compile(r'^(\s+)rev:(\s*)([^\s#]+)(.*)$', re.DOTALL) -REV_LINE_FMT = '{}rev:{}{}{}' +REV_LINE_RE = re.compile(r'^(\s+)rev:(\s*)([\'"]?)([^\s#]+)(.*)(\r?\n)$') -def _write_new_config_file(path, output): - with open(path) as f: - original_contents = f.read() - output = remove_defaults(output, CONFIG_SCHEMA) - new_contents = ordered_dump(output, **C.YAML_DUMP_KWARGS) +def _original_lines( + path: str, + rev_infos: list[RevInfo | None], + retry: bool = False, +) -> tuple[list[str], list[int]]: + """detect `rev:` lines or reformat the file""" + with open(path, newline='') as f: + original = f.read() - lines = original_contents.splitlines(True) - rev_line_indices_reversed = list( - reversed([ - i for i, line in enumerate(lines) if REV_LINE_RE.match(line) - ]), - ) - - for line in new_contents.splitlines(True): - if REV_LINE_RE.match(line): - # It's possible we didn't identify the rev lines in the original - if not rev_line_indices_reversed: - break - line_index = rev_line_indices_reversed.pop() - original_line = lines[line_index] - orig_match = REV_LINE_RE.match(original_line) - new_match = REV_LINE_RE.match(line) - lines[line_index] = REV_LINE_FMT.format( - orig_match.group(1), orig_match.group(2), - new_match.group(3), orig_match.group(4), - ) - - # If we failed to intelligently rewrite the rev lines, fall back to the - # pretty-formatted yaml output - to_write = ''.join(lines) - if remove_defaults(ordered_load(to_write), CONFIG_SCHEMA) != output: - to_write = new_contents - - with open(path, 'w') as f: - f.write(to_write) + lines = original.splitlines(True) + idxs = [i for i, line in enumerate(lines) if REV_LINE_RE.match(line)] + if len(idxs) == len(rev_infos): + return lines, idxs + elif retry: + raise AssertionError('could not find rev lines') + else: + with open(path, 'w') as f: + f.write(yaml_dump(yaml_load(original))) + return _original_lines(path, rev_infos, retry=True) -def autoupdate(config_file, store, tags_only, repos=()): +def _write_new_config(path: str, rev_infos: list[RevInfo | None]) -> None: + lines, idxs = _original_lines(path, rev_infos) + + for idx, rev_info in zip(idxs, rev_infos): + if rev_info is None: + continue + match = REV_LINE_RE.match(lines[idx]) + assert match is not None + new_rev_s = yaml_dump({'rev': rev_info.rev}, default_style=match[3]) + new_rev = new_rev_s.split(':', 1)[1].strip() + if rev_info.frozen is not None: + comment = f' # frozen: {rev_info.frozen}' + elif match[5].strip().startswith('# frozen:'): + comment = '' + else: + comment = match[5] + lines[idx] = f'{match[1]}rev:{match[2]}{new_rev}{comment}{match[6]}' + + with open(path, 'w', newline='') as f: + f.write(''.join(lines)) + + +def autoupdate( + config_file: str, + tags_only: bool, + freeze: bool, + repos: Sequence[str] = (), + jobs: int = 1, +) -> int: """Auto-update the pre-commit config to the latest versions of repos.""" migrate_config(config_file, quiet=True) - retv = 0 - output_repos = [] changed = False + retv = 0 - input_config = load_config(config_file) + config_repos = [ + repo for repo in load_config(config_file)['repos'] + if repo['repo'] not in {LOCAL, META} + ] - for repo_config in input_config['repos']: - if ( - repo_config['repo'] in {LOCAL, META} or - # Skip updating any repo_configs that aren't for the specified repo - repos and repo_config['repo'] not in repos - ): - output_repos.append(repo_config) - continue - output.write('Updating {}...'.format(repo_config['repo'])) - try: - new_repo_config = _update_repo(repo_config, store, tags_only) - except RepositoryCannotBeUpdatedError as error: - output.write_line(error.args[0]) - output_repos.append(repo_config) - retv = 1 - continue - - if new_repo_config['rev'] != repo_config['rev']: - changed = True - output.write_line( - 'updating {} -> {}.'.format( - repo_config['rev'], new_repo_config['rev'], - ), + rev_infos: list[RevInfo | None] = [None] * len(config_repos) + jobs = jobs or xargs.cpu_count() # 0 => number of cpus + jobs = min(jobs, len(repos) or len(config_repos)) # max 1-per-thread + jobs = max(jobs, 1) # at least one thread + with concurrent.futures.ThreadPoolExecutor(jobs) as exe: + futures = [ + exe.submit( + _update_one, + i, repo, tags_only=tags_only, freeze=freeze, ) - output_repos.append(new_repo_config) - else: - output.write_line('already up to date.') - output_repos.append(repo_config) + for i, repo in enumerate(config_repos) + if not repos or repo['repo'] in repos + ] + for future in concurrent.futures.as_completed(futures): + try: + i, old, new = future.result() + except RepositoryCannotBeUpdatedError as e: + output.write_line(str(e)) + retv = 1 + else: + if new.rev != old.rev: + changed = True + if new.frozen: + new_s = f'{new.frozen} (frozen)' + else: + new_s = new.rev + msg = f'updating {old.rev} -> {new_s}' + rev_infos[i] = new + else: + msg = 'already up to date!' + + output.write_line(f'[{old.repo}] {msg}') if changed: - output_config = input_config.copy() - output_config['repos'] = output_repos - _write_new_config_file(config_file, output_config) + _write_new_config(config_file, rev_infos) return retv diff --git a/pre_commit/commands/clean.py b/pre_commit/commands/clean.py index 5c763029..5119f645 100644 --- a/pre_commit/commands/clean.py +++ b/pre_commit/commands/clean.py @@ -1,16 +1,16 @@ -from __future__ import print_function -from __future__ import unicode_literals +from __future__ import annotations import os.path from pre_commit import output +from pre_commit.store import Store from pre_commit.util import rmtree -def clean(store): +def clean(store: Store) -> int: legacy_path = os.path.expanduser('~/.pre-commit') for directory in (store.directory, legacy_path): if os.path.exists(directory): rmtree(directory) - output.write_line('Cleaned {}.'.format(directory)) + output.write_line(f'Cleaned {directory}.') return 0 diff --git a/pre_commit/commands/gc.py b/pre_commit/commands/gc.py index 65818e50..975d5e4c 100644 --- a/pre_commit/commands/gc.py +++ b/pre_commit/commands/gc.py @@ -1,7 +1,7 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os.path +from typing import Any import pre_commit.constants as C from pre_commit import output @@ -11,16 +11,24 @@ from pre_commit.clientlib import load_config from pre_commit.clientlib import load_manifest from pre_commit.clientlib import LOCAL from pre_commit.clientlib import META +from pre_commit.store import Store +from pre_commit.util import rmtree -def _mark_used_repos(store, all_repos, unused_repos, repo): +def _mark_used_repos( + store: Store, + all_repos: dict[tuple[str, str], str], + unused_repos: set[tuple[str, str]], + repo: dict[str, Any], +) -> None: if repo['repo'] == META: return elif repo['repo'] == LOCAL: for hook in repo['hooks']: deps = hook.get('additional_dependencies') unused_repos.discard(( - store.db_repo_name(repo['repo'], deps), C.LOCAL_REPO_VERSION, + store.db_repo_name(repo['repo'], deps), + C.LOCAL_REPO_VERSION, )) else: key = (repo['repo'], repo['rev']) @@ -50,34 +58,41 @@ def _mark_used_repos(store, all_repos, unused_repos, repo): )) -def _gc_repos(store): - configs = store.select_all_configs() - repos = store.select_all_repos() +def _gc(store: Store) -> int: + with store.exclusive_lock(), store.connect() as db: + store._create_configs_table(db) - # delete config paths which do not exist - dead_configs = [p for p in configs if not os.path.exists(p)] - live_configs = [p for p in configs if os.path.exists(p)] + repos = db.execute('SELECT repo, ref, path FROM repos').fetchall() + all_repos = {(repo, ref): path for repo, ref, path in repos} + unused_repos = set(all_repos) - all_repos = {(repo, ref): path for repo, ref, path in repos} - unused_repos = set(all_repos) - for config_path in live_configs: - try: - config = load_config(config_path) - except InvalidConfigError: - dead_configs.append(config_path) - continue - else: - for repo in config['repos']: - _mark_used_repos(store, all_repos, unused_repos, repo) + configs_rows = db.execute('SELECT path FROM configs').fetchall() + configs = [path for path, in configs_rows] - store.delete_configs(dead_configs) - for db_repo_name, ref in unused_repos: - store.delete_repo(db_repo_name, ref, all_repos[(db_repo_name, ref)]) - return len(unused_repos) + dead_configs = [] + for config_path in configs: + try: + config = load_config(config_path) + except InvalidConfigError: + dead_configs.append(config_path) + continue + else: + for repo in config['repos']: + _mark_used_repos(store, all_repos, unused_repos, repo) + + paths = [(path,) for path in dead_configs] + db.executemany('DELETE FROM configs WHERE path = ?', paths) + + db.executemany( + 'DELETE FROM repos WHERE repo = ? and ref = ?', + sorted(unused_repos), + ) + for k in unused_repos: + rmtree(all_repos[k]) + + return len(unused_repos) -def gc(store): - with store.exclusive_lock(): - repos_removed = _gc_repos(store) - output.write_line('{} repo(s) removed.'.format(repos_removed)) +def gc(store: Store) -> int: + output.write_line(f'{_gc(store)} repo(s) removed.') return 0 diff --git a/pre_commit/commands/hazmat.py b/pre_commit/commands/hazmat.py new file mode 100644 index 00000000..01b27ce6 --- /dev/null +++ b/pre_commit/commands/hazmat.py @@ -0,0 +1,95 @@ +from __future__ import annotations + +import argparse +import subprocess +from collections.abc import Sequence + +from pre_commit.parse_shebang import normalize_cmd + + +def add_parsers(parser: argparse.ArgumentParser) -> None: + subparsers = parser.add_subparsers(dest='tool') + + cd_parser = subparsers.add_parser( + 'cd', help='cd to a subdir and run the command', + ) + cd_parser.add_argument('subdir') + cd_parser.add_argument('cmd', nargs=argparse.REMAINDER) + + ignore_exit_code_parser = subparsers.add_parser( + 'ignore-exit-code', help='run the command but ignore the exit code', + ) + ignore_exit_code_parser.add_argument('cmd', nargs=argparse.REMAINDER) + + n1_parser = subparsers.add_parser( + 'n1', help='run the command once per filename', + ) + n1_parser.add_argument('cmd', nargs=argparse.REMAINDER) + + +def _cmd_filenames(cmd: tuple[str, ...]) -> tuple[ + tuple[str, ...], + tuple[str, ...], +]: + for idx, val in enumerate(reversed(cmd)): + if val == '--': + split = len(cmd) - idx + break + else: + raise SystemExit('hazmat entry must end with `--`') + + return cmd[:split - 1], cmd[split:] + + +def cd(subdir: str, cmd: tuple[str, ...]) -> int: + cmd, filenames = _cmd_filenames(cmd) + + prefix = f'{subdir}/' + new_filenames = [] + for filename in filenames: + if not filename.startswith(prefix): + raise SystemExit(f'unexpected file without {prefix=}: {filename}') + else: + new_filenames.append(filename.removeprefix(prefix)) + + cmd = normalize_cmd(cmd) + return subprocess.call((*cmd, *new_filenames), cwd=subdir) + + +def ignore_exit_code(cmd: tuple[str, ...]) -> int: + cmd = normalize_cmd(cmd) + subprocess.call(cmd) + return 0 + + +def n1(cmd: tuple[str, ...]) -> int: + cmd, filenames = _cmd_filenames(cmd) + cmd = normalize_cmd(cmd) + ret = 0 + for filename in filenames: + ret |= subprocess.call((*cmd, filename)) + return ret + + +def impl(args: argparse.Namespace) -> int: + args.cmd = tuple(args.cmd) + if args.tool == 'cd': + return cd(args.subdir, args.cmd) + elif args.tool == 'ignore-exit-code': + return ignore_exit_code(args.cmd) + elif args.tool == 'n1': + return n1(args.cmd) + else: + raise NotImplementedError(f'unexpected tool: {args.tool}') + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser() + add_parsers(parser) + args = parser.parse_args(argv) + + return impl(args) + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/pre_commit/commands/hook_impl.py b/pre_commit/commands/hook_impl.py new file mode 100644 index 00000000..de5c8f34 --- /dev/null +++ b/pre_commit/commands/hook_impl.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import argparse +import os.path +import subprocess +import sys +from collections.abc import Sequence + +from pre_commit.commands.run import run +from pre_commit.envcontext import envcontext +from pre_commit.parse_shebang import normalize_cmd +from pre_commit.store import Store + +Z40 = '0' * 40 + + +def _run_legacy( + hook_type: str, + hook_dir: str, + args: Sequence[str], +) -> tuple[int, bytes]: + if os.environ.get('PRE_COMMIT_RUNNING_LEGACY'): + raise SystemExit( + f"bug: pre-commit's script is installed in migration mode\n" + f'run `pre-commit install -f --hook-type {hook_type}` to fix ' + f'this\n\n' + f'Please report this bug at ' + f'https://github.com/pre-commit/pre-commit/issues', + ) + + if hook_type == 'pre-push': + stdin = sys.stdin.buffer.read() + else: + stdin = b'' + + # not running in legacy mode + legacy_hook = os.path.join(hook_dir, f'{hook_type}.legacy') + if not os.access(legacy_hook, os.X_OK): + return 0, stdin + + with envcontext((('PRE_COMMIT_RUNNING_LEGACY', '1'),)): + cmd = normalize_cmd((legacy_hook, *args)) + return subprocess.run(cmd, input=stdin).returncode, stdin + + +def _validate_config( + retv: int, + config: str, + skip_on_missing_config: bool, +) -> None: + if not os.path.isfile(config): + if skip_on_missing_config or os.getenv('PRE_COMMIT_ALLOW_NO_CONFIG'): + print(f'`{config}` config file not found. Skipping `pre-commit`.') + raise SystemExit(retv) + else: + print( + f'No {config} file was found\n' + f'- To temporarily silence this, run ' + f'`PRE_COMMIT_ALLOW_NO_CONFIG=1 git ...`\n' + f'- To permanently silence this, install pre-commit with the ' + f'--allow-missing-config option\n' + f'- To uninstall pre-commit run `pre-commit uninstall`', + ) + raise SystemExit(1) + + +def _ns( + hook_type: str, + color: bool, + *, + all_files: bool = False, + remote_branch: str | None = None, + local_branch: str | None = None, + from_ref: str | None = None, + to_ref: str | None = None, + pre_rebase_upstream: str | None = None, + pre_rebase_branch: str | None = None, + remote_name: str | None = None, + remote_url: str | None = None, + commit_msg_filename: str | None = None, + prepare_commit_message_source: str | None = None, + commit_object_name: str | None = None, + checkout_type: str | None = None, + is_squash_merge: str | None = None, + rewrite_command: str | None = None, +) -> argparse.Namespace: + return argparse.Namespace( + color=color, + hook_stage=hook_type, + remote_branch=remote_branch, + local_branch=local_branch, + from_ref=from_ref, + to_ref=to_ref, + pre_rebase_upstream=pre_rebase_upstream, + pre_rebase_branch=pre_rebase_branch, + remote_name=remote_name, + remote_url=remote_url, + commit_msg_filename=commit_msg_filename, + prepare_commit_message_source=prepare_commit_message_source, + commit_object_name=commit_object_name, + all_files=all_files, + checkout_type=checkout_type, + is_squash_merge=is_squash_merge, + rewrite_command=rewrite_command, + files=(), + hook=None, + verbose=False, + show_diff_on_failure=False, + fail_fast=False, + ) + + +def _rev_exists(rev: str) -> bool: + return not subprocess.call(('git', 'rev-list', '--quiet', rev)) + + +def _pre_push_ns( + color: bool, + args: Sequence[str], + stdin: bytes, +) -> argparse.Namespace | None: + remote_name = args[0] + remote_url = args[1] + + for line in stdin.decode().splitlines(): + parts = line.rsplit(maxsplit=3) + local_branch, local_sha, remote_branch, remote_sha = parts + if local_sha == Z40: + continue + elif remote_sha != Z40 and _rev_exists(remote_sha): + return _ns( + 'pre-push', color, + from_ref=remote_sha, to_ref=local_sha, + remote_branch=remote_branch, + local_branch=local_branch, + remote_name=remote_name, remote_url=remote_url, + ) + else: + # ancestors not found in remote + ancestors = subprocess.check_output(( + 'git', 'rev-list', local_sha, '--topo-order', '--reverse', + '--not', f'--remotes={remote_name}', + )).decode().strip() + if not ancestors: + continue + else: + first_ancestor = ancestors.splitlines()[0] + cmd = ('git', 'rev-list', '--max-parents=0', local_sha) + roots = set(subprocess.check_output(cmd).decode().splitlines()) + if first_ancestor in roots: + # pushing the whole tree including root commit + return _ns( + 'pre-push', color, + all_files=True, + remote_name=remote_name, remote_url=remote_url, + remote_branch=remote_branch, + local_branch=local_branch, + ) + else: + rev_cmd = ('git', 'rev-parse', f'{first_ancestor}^') + source = subprocess.check_output(rev_cmd).decode().strip() + return _ns( + 'pre-push', color, + from_ref=source, to_ref=local_sha, + remote_name=remote_name, remote_url=remote_url, + remote_branch=remote_branch, + local_branch=local_branch, + ) + + # nothing to push + return None + + +_EXPECTED_ARG_LENGTH_BY_HOOK = { + 'commit-msg': 1, + 'post-checkout': 3, + 'post-commit': 0, + 'pre-commit': 0, + 'pre-merge-commit': 0, + 'post-merge': 1, + 'post-rewrite': 1, + 'pre-push': 2, +} + + +def _check_args_length(hook_type: str, args: Sequence[str]) -> None: + if hook_type == 'prepare-commit-msg': + if len(args) < 1 or len(args) > 3: + raise SystemExit( + f'hook-impl for {hook_type} expected 1, 2, or 3 arguments ' + f'but got {len(args)}: {args}', + ) + elif hook_type == 'pre-rebase': + if len(args) < 1 or len(args) > 2: + raise SystemExit( + f'hook-impl for {hook_type} expected 1 or 2 arguments ' + f'but got {len(args)}: {args}', + ) + elif hook_type in _EXPECTED_ARG_LENGTH_BY_HOOK: + expected = _EXPECTED_ARG_LENGTH_BY_HOOK[hook_type] + if len(args) != expected: + arguments_s = 'argument' if expected == 1 else 'arguments' + raise SystemExit( + f'hook-impl for {hook_type} expected {expected} {arguments_s} ' + f'but got {len(args)}: {args}', + ) + else: + raise AssertionError(f'unexpected hook type: {hook_type}') + + +def _run_ns( + hook_type: str, + color: bool, + args: Sequence[str], + stdin: bytes, +) -> argparse.Namespace | None: + _check_args_length(hook_type, args) + if hook_type == 'pre-push': + return _pre_push_ns(color, args, stdin) + elif hook_type in 'commit-msg': + return _ns(hook_type, color, commit_msg_filename=args[0]) + elif hook_type == 'prepare-commit-msg' and len(args) == 1: + return _ns(hook_type, color, commit_msg_filename=args[0]) + elif hook_type == 'prepare-commit-msg' and len(args) == 2: + return _ns( + hook_type, color, commit_msg_filename=args[0], + prepare_commit_message_source=args[1], + ) + elif hook_type == 'prepare-commit-msg' and len(args) == 3: + return _ns( + hook_type, color, commit_msg_filename=args[0], + prepare_commit_message_source=args[1], commit_object_name=args[2], + ) + elif hook_type in {'post-commit', 'pre-merge-commit', 'pre-commit'}: + return _ns(hook_type, color) + elif hook_type == 'post-checkout': + return _ns( + hook_type, color, + from_ref=args[0], to_ref=args[1], checkout_type=args[2], + ) + elif hook_type == 'post-merge': + return _ns(hook_type, color, is_squash_merge=args[0]) + elif hook_type == 'post-rewrite': + return _ns(hook_type, color, rewrite_command=args[0]) + elif hook_type == 'pre-rebase' and len(args) == 1: + return _ns(hook_type, color, pre_rebase_upstream=args[0]) + elif hook_type == 'pre-rebase' and len(args) == 2: + return _ns( + hook_type, color, pre_rebase_upstream=args[0], + pre_rebase_branch=args[1], + ) + else: + raise AssertionError(f'unexpected hook type: {hook_type}') + + +def hook_impl( + store: Store, + *, + config: str, + color: bool, + hook_type: str, + hook_dir: str, + skip_on_missing_config: bool, + args: Sequence[str], +) -> int: + retv, stdin = _run_legacy(hook_type, hook_dir, args) + _validate_config(retv, config, skip_on_missing_config) + ns = _run_ns(hook_type, color, args, stdin) + if ns is None: + return retv + else: + return retv | run(config, store, ns) diff --git a/pre_commit/commands/init_templatedir.py b/pre_commit/commands/init_templatedir.py index 6e8df18c..08af6561 100644 --- a/pre_commit/commands/init_templatedir.py +++ b/pre_commit/commands/init_templatedir.py @@ -1,17 +1,30 @@ +from __future__ import annotations + import logging import os.path from pre_commit.commands.install_uninstall import install +from pre_commit.store import Store from pre_commit.util import CalledProcessError from pre_commit.util import cmd_output logger = logging.getLogger('pre_commit') -def init_templatedir(config_file, store, directory, hook_type): +def init_templatedir( + config_file: str, + store: Store, + directory: str, + hook_types: list[str] | None, + skip_on_missing_config: bool = True, +) -> int: install( - config_file, store, overwrite=True, hook_type=hook_type, - skip_on_missing_config=True, git_dir=directory, + config_file, + store, + hook_types=hook_types, + overwrite=True, + skip_on_missing_config=skip_on_missing_config, + git_dir=directory, ) try: _, out, _ = cmd_output('git', 'config', 'init.templateDir') @@ -22,6 +35,5 @@ def init_templatedir(config_file, store, directory, hook_type): dest = os.path.realpath(directory) if configured_path != dest: logger.warning('`init.templateDir` not set to the target directory') - logger.warning( - 'maybe `git config --global init.templateDir {}`?'.format(dest), - ) + logger.warning(f'maybe `git config --global init.templateDir {dest}`?') + return 0 diff --git a/pre_commit/commands/install_uninstall.py b/pre_commit/commands/install_uninstall.py index 9b2c3b80..d19e0d47 100644 --- a/pre_commit/commands/install_uninstall.py +++ b/pre_commit/commands/install_uninstall.py @@ -1,21 +1,19 @@ -from __future__ import print_function -from __future__ import unicode_literals +from __future__ import annotations -import io -import itertools import logging import os.path +import shlex import shutil import sys from pre_commit import git from pre_commit import output +from pre_commit.clientlib import InvalidConfigError from pre_commit.clientlib import load_config from pre_commit.repository import all_hooks from pre_commit.repository import install_hook_envs -from pre_commit.util import cmd_output +from pre_commit.store import Store from pre_commit.util import make_executable -from pre_commit.util import mkdirp from pre_commit.util import resource_text @@ -23,66 +21,56 @@ logger = logging.getLogger(__name__) # This is used to identify the hook file we install PRIOR_HASHES = ( - '4d9958c90bc262f47553e2c073f14cfe', - 'd8ee923c46731b42cd95cc869add4062', - '49fd668cb42069aa1b6048464be5d395', - '79f09a650522a87b0da915d0d983b2de', - 'e358c9dae00eac5d06b38dfdb1e33a8c', + b'4d9958c90bc262f47553e2c073f14cfe', + b'd8ee923c46731b42cd95cc869add4062', + b'49fd668cb42069aa1b6048464be5d395', + b'79f09a650522a87b0da915d0d983b2de', + b'e358c9dae00eac5d06b38dfdb1e33a8c', ) -CURRENT_HASH = '138fd403232d2ddd5efb44317e38bf03' +CURRENT_HASH = b'138fd403232d2ddd5efb44317e38bf03' TEMPLATE_START = '# start templated\n' TEMPLATE_END = '# end templated\n' -def _hook_paths(hook_type, git_dir=None): - git_dir = git_dir if git_dir is not None else git.get_git_dir() +def _hook_types(cfg_filename: str, hook_types: list[str] | None) -> list[str]: + if hook_types is not None: + return hook_types + else: + try: + cfg = load_config(cfg_filename) + except InvalidConfigError: + return ['pre-commit'] + else: + return cfg['default_install_hook_types'] + + +def _hook_paths( + hook_type: str, + git_dir: str | None = None, +) -> tuple[str, str]: + git_dir = git_dir if git_dir is not None else git.get_git_common_dir() pth = os.path.join(git_dir, 'hooks', hook_type) - return pth, '{}.legacy'.format(pth) + return pth, f'{pth}.legacy' -def is_our_script(filename): - if not os.path.exists(filename): # pragma: windows no cover (symlink) +def is_our_script(filename: str) -> bool: + if not os.path.exists(filename): # pragma: win32 no cover (symlink) return False - with io.open(filename) as f: + with open(filename, 'rb') as f: contents = f.read() return any(h in contents for h in (CURRENT_HASH,) + PRIOR_HASHES) -def shebang(): - if sys.platform == 'win32': - py = 'python' - else: - # Homebrew/homebrew-core#35825: be more timid about appropriate `PATH` - path_choices = [p for p in os.defpath.split(os.pathsep) if p] - exe_choices = [ - 'python{}'.format('.'.join(str(v) for v in sys.version_info[:i])) - for i in range(3) - ] - for path, exe in itertools.product(path_choices, exe_choices): - if os.path.exists(os.path.join(path, exe)): - py = exe - break - else: - py = 'python' - return '#!/usr/bin/env {}'.format(py) - - -def install( - config_file, store, - overwrite=False, hooks=False, hook_type='pre-commit', - skip_on_missing_config=False, git_dir=None, -): - """Install the pre-commit hooks.""" - if cmd_output('git', 'config', 'core.hooksPath', retcode=None)[1].strip(): - logger.error( - 'Cowardly refusing to install hooks with `core.hooksPath` set.\n' - 'hint: `git config --unset-all core.hooksPath`', - ) - return 1 - +def _install_hook_script( + config_file: str, + hook_type: str, + overwrite: bool = False, + skip_on_missing_config: bool = False, + git_dir: str | None = None, +) -> None: hook_path, legacy_path = _hook_paths(hook_type, git_dir=git_dir) - mkdirp(os.path.dirname(hook_path)) + os.makedirs(os.path.dirname(hook_path), exist_ok=True) # If we have an existing hook, move it to pre-commit.legacy if os.path.lexists(hook_path) and not is_our_script(hook_path): @@ -93,57 +81,87 @@ def install( os.remove(legacy_path) elif os.path.exists(legacy_path): output.write_line( - 'Running in migration mode with existing hooks at {}\n' - 'Use -f to use only pre-commit.'.format(legacy_path), + f'Running in migration mode with existing hooks at {legacy_path}\n' + f'Use -f to use only pre-commit.', ) - params = { - 'CONFIG': config_file, - 'HOOK_TYPE': hook_type, - 'INSTALL_PYTHON': sys.executable, - 'SKIP_ON_MISSING_CONFIG': skip_on_missing_config, - } + args = ['hook-impl', f'--config={config_file}', f'--hook-type={hook_type}'] + if skip_on_missing_config: + args.append('--skip-on-missing-config') - with io.open(hook_path, 'w') as hook_file: + with open(hook_path, 'w') as hook_file: contents = resource_text('hook-tmpl') before, rest = contents.split(TEMPLATE_START) - to_template, after = rest.split(TEMPLATE_END) + _, after = rest.split(TEMPLATE_END) - before = before.replace('#!/usr/bin/env python3', shebang()) + # on windows always use `/bin/sh` since `bash` might not be on PATH + # though we use bash-specific features `sh` on windows is actually + # bash in "POSIXLY_CORRECT" mode which still supports the features we + # use: subshells / arrays + if sys.platform == 'win32': # pragma: win32 cover + hook_file.write('#!/bin/sh\n') hook_file.write(before + TEMPLATE_START) - for line in to_template.splitlines(): - var = line.split()[0] - hook_file.write('{} = {!r}\n'.format(var, params[var])) + hook_file.write(f'INSTALL_PYTHON={shlex.quote(sys.executable)}\n') + args_s = shlex.join(args) + hook_file.write(f'ARGS=({args_s})\n') hook_file.write(TEMPLATE_END + after) make_executable(hook_path) - output.write_line('pre-commit installed at {}'.format(hook_path)) + output.write_line(f'pre-commit installed at {hook_path}') + + +def install( + config_file: str, + store: Store, + hook_types: list[str] | None, + overwrite: bool = False, + hooks: bool = False, + skip_on_missing_config: bool = False, + git_dir: str | None = None, +) -> int: + if git_dir is None and git.has_core_hookpaths_set(): + logger.error( + 'Cowardly refusing to install hooks with `core.hooksPath` set.\n' + 'hint: `git config --unset-all core.hooksPath`', + ) + return 1 + + for hook_type in _hook_types(config_file, hook_types): + _install_hook_script( + config_file, hook_type, + overwrite=overwrite, + skip_on_missing_config=skip_on_missing_config, + git_dir=git_dir, + ) - # If they requested we install all of the hooks, do so. if hooks: install_hooks(config_file, store) return 0 -def install_hooks(config_file, store): +def install_hooks(config_file: str, store: Store) -> int: install_hook_envs(all_hooks(load_config(config_file), store), store) + return 0 -def uninstall(hook_type='pre-commit'): - """Uninstall the pre-commit hooks.""" +def _uninstall_hook_script(hook_type: str) -> None: hook_path, legacy_path = _hook_paths(hook_type) # If our file doesn't exist or it isn't ours, gtfo. if not os.path.exists(hook_path) or not is_our_script(hook_path): - return 0 + return os.remove(hook_path) - output.write_line('{} uninstalled'.format(hook_type)) + output.write_line(f'{hook_type} uninstalled') if os.path.exists(legacy_path): - os.rename(legacy_path, hook_path) - output.write_line('Restored previous hooks to {}'.format(hook_path)) + os.replace(legacy_path, hook_path) + output.write_line(f'Restored previous hooks to {hook_path}') + +def uninstall(config_file: str, hook_types: list[str] | None) -> int: + for hook_type in _hook_types(config_file, hook_types): + _uninstall_hook_script(hook_type) return 0 diff --git a/pre_commit/commands/migrate_config.py b/pre_commit/commands/migrate_config.py index bac42319..b04c53a5 100644 --- a/pre_commit/commands/migrate_config.py +++ b/pre_commit/commands/migrate_config.py @@ -1,62 +1,135 @@ -from __future__ import print_function -from __future__ import unicode_literals +from __future__ import annotations -import io -import re +import functools +import itertools +import textwrap +from collections.abc import Callable +import cfgv import yaml -from aspy.yaml import ordered_load +from yaml.nodes import ScalarNode + +from pre_commit.clientlib import InvalidConfigError +from pre_commit.yaml import yaml_compose +from pre_commit.yaml import yaml_load +from pre_commit.yaml_rewrite import MappingKey +from pre_commit.yaml_rewrite import MappingValue +from pre_commit.yaml_rewrite import match +from pre_commit.yaml_rewrite import SequenceItem -def _indent(s): - lines = s.splitlines(True) - return ''.join(' ' * 4 + line if line.strip() else line for line in lines) +def _is_header_line(line: str) -> bool: + return line.startswith(('#', '---')) or not line.strip() -def _is_header_line(line): - return (line.startswith(('#', '---')) or not line.strip()) +def _migrate_map(contents: str) -> str: + if isinstance(yaml_load(contents), list): + # Find the first non-header line + lines = contents.splitlines(True) + i = 0 + # Only loop on non empty configuration file + while i < len(lines) and _is_header_line(lines[i]): + i += 1 + header = ''.join(lines[:i]) + rest = ''.join(lines[i:]) -def _migrate_map(contents): - # Find the first non-header line - lines = contents.splitlines(True) - i = 0 - # Only loop on non empty configuration file - while i < len(lines) and _is_header_line(lines[i]): - i += 1 - - header = ''.join(lines[:i]) - rest = ''.join(lines[i:]) - - if isinstance(ordered_load(contents), list): # If they are using the "default" flow style of yaml, this operation # will yield a valid configuration try: - trial_contents = header + 'repos:\n' + rest - ordered_load(trial_contents) + trial_contents = f'{header}repos:\n{rest}' + yaml_load(trial_contents) contents = trial_contents except yaml.YAMLError: - contents = header + 'repos:\n' + _indent(rest) + contents = f'{header}repos:\n{textwrap.indent(rest, " " * 4)}' return contents -def _migrate_sha_to_rev(contents): - reg = re.compile(r'(\n\s+)sha:') - return reg.sub(r'\1rev:', contents) +def _preserve_style(n: ScalarNode, *, s: str) -> str: + style = n.style or '' + return f'{style}{s}{style}' -def migrate_config(config_file, quiet=False): - with io.open(config_file) as f: +def _fix_stage(n: ScalarNode) -> str: + return _preserve_style(n, s=f'pre-{n.value}') + + +def _migrate_composed(contents: str) -> str: + tree = yaml_compose(contents) + rewrites: list[tuple[ScalarNode, Callable[[ScalarNode], str]]] = [] + + # sha -> rev + sha_to_rev_replace = functools.partial(_preserve_style, s='rev') + sha_to_rev_matcher = ( + MappingValue('repos'), + SequenceItem(), + MappingKey('sha'), + ) + for node in match(tree, sha_to_rev_matcher): + rewrites.append((node, sha_to_rev_replace)) + + # python_venv -> python + language_matcher = ( + MappingValue('repos'), + SequenceItem(), + MappingValue('hooks'), + SequenceItem(), + MappingValue('language'), + ) + python_venv_replace = functools.partial(_preserve_style, s='python') + for node in match(tree, language_matcher): + if node.value == 'python_venv': + rewrites.append((node, python_venv_replace)) + + # stages rewrites + default_stages_matcher = (MappingValue('default_stages'), SequenceItem()) + default_stages_match = match(tree, default_stages_matcher) + hook_stages_matcher = ( + MappingValue('repos'), + SequenceItem(), + MappingValue('hooks'), + SequenceItem(), + MappingValue('stages'), + SequenceItem(), + ) + hook_stages_match = match(tree, hook_stages_matcher) + for node in itertools.chain(default_stages_match, hook_stages_match): + if node.value in {'commit', 'push', 'merge-commit'}: + rewrites.append((node, _fix_stage)) + + rewrites.sort(reverse=True, key=lambda nf: nf[0].start_mark.index) + + src_parts = [] + end: int | None = None + for node, func in rewrites: + src_parts.append(contents[node.end_mark.index:end]) + src_parts.append(func(node)) + end = node.start_mark.index + src_parts.append(contents[:end]) + src_parts.reverse() + return ''.join(src_parts) + + +def migrate_config(config_file: str, quiet: bool = False) -> int: + with open(config_file) as f: orig_contents = contents = f.read() + with cfgv.reraise_as(InvalidConfigError): + with cfgv.validate_context(f'File {config_file}'): + try: + yaml_load(orig_contents) + except Exception as e: + raise cfgv.ValidationError(str(e)) + contents = _migrate_map(contents) - contents = _migrate_sha_to_rev(contents) + contents = _migrate_composed(contents) if contents != orig_contents: - with io.open(config_file, 'w') as f: + with open(config_file, 'w') as f: f.write(contents) print('Configuration has been migrated.') elif not quiet: print('Configuration is already migrated.') + return 0 diff --git a/pre_commit/commands/run.py b/pre_commit/commands/run.py index 4087a650..8ab505ff 100644 --- a/pre_commit/commands/run.py +++ b/pre_commit/commands/run.py @@ -1,185 +1,248 @@ -from __future__ import unicode_literals +from __future__ import annotations +import argparse +import contextlib +import functools import logging import os import re import subprocess -import sys +import time +import unicodedata +from collections.abc import Generator +from collections.abc import Iterable +from collections.abc import MutableMapping +from collections.abc import Sequence +from typing import Any from identify.identify import tags_from_path from pre_commit import color from pre_commit import git from pre_commit import output +from pre_commit.all_languages import languages from pre_commit.clientlib import load_config -from pre_commit.output import get_hook_message +from pre_commit.hook import Hook from pre_commit.repository import all_hooks from pre_commit.repository import install_hook_envs from pre_commit.staged_files_only import staged_files_only -from pre_commit.util import cmd_output -from pre_commit.util import noop_context +from pre_commit.store import Store +from pre_commit.util import cmd_output_b logger = logging.getLogger('pre_commit') -def filter_by_include_exclude(names, include, exclude): +def _len_cjk(msg: str) -> int: + widths = {'A': 1, 'F': 2, 'H': 1, 'N': 1, 'Na': 1, 'W': 2} + return sum(widths[unicodedata.east_asian_width(c)] for c in msg) + + +def _start_msg(*, start: str, cols: int, end_len: int) -> str: + dots = '.' * (cols - _len_cjk(start) - end_len - 1) + return f'{start}{dots}' + + +def _full_msg( + *, + start: str, + cols: int, + end_msg: str, + end_color: str, + use_color: bool, + postfix: str = '', +) -> str: + dots = '.' * (cols - _len_cjk(start) - len(postfix) - len(end_msg) - 1) + end = color.format_color(end_msg, end_color, use_color) + return f'{start}{dots}{postfix}{end}\n' + + +def filter_by_include_exclude( + names: Iterable[str], + include: str, + exclude: str, +) -> Generator[str]: include_re, exclude_re = re.compile(include), re.compile(exclude) - return [ + return ( filename for filename in names if include_re.search(filename) if not exclude_re.search(filename) - ] + ) -class Classifier(object): - def __init__(self, filenames): +class Classifier: + def __init__(self, filenames: Iterable[str]) -> None: self.filenames = [f for f in filenames if os.path.lexists(f)] - self._types_cache = {} - def _types_for_file(self, filename): - try: - return self._types_cache[filename] - except KeyError: - ret = self._types_cache[filename] = tags_from_path(filename) - return ret + @functools.cache + def _types_for_file(self, filename: str) -> set[str]: + return tags_from_path(filename) - def by_types(self, names, types, exclude_types): - types, exclude_types = frozenset(types), frozenset(exclude_types) - ret = [] + def by_types( + self, + names: Iterable[str], + types: Iterable[str], + types_or: Iterable[str], + exclude_types: Iterable[str], + ) -> Generator[str]: + types = frozenset(types) + types_or = frozenset(types_or) + exclude_types = frozenset(exclude_types) for filename in names: tags = self._types_for_file(filename) - if tags >= types and not tags & exclude_types: - ret.append(filename) - return ret + if ( + tags >= types and + (not types_or or tags & types_or) and + not tags & exclude_types + ): + yield filename - def filenames_for_hook(self, hook): - names = self.filenames - names = filter_by_include_exclude(names, hook.files, hook.exclude) - names = self.by_types(names, hook.types, hook.exclude_types) - return names + def filenames_for_hook(self, hook: Hook) -> Generator[str]: + return self.by_types( + filter_by_include_exclude( + self.filenames, + hook.files, + hook.exclude, + ), + hook.types, + hook.types_or, + hook.exclude_types, + ) + + @classmethod + def from_config( + cls, + filenames: Iterable[str], + include: str, + exclude: str, + ) -> Classifier: + # on windows we normalize all filenames to use forward slashes + # this makes it easier to filter using the `files:` regex + # this also makes improperly quoted shell-based hooks work better + # see #1173 + if os.altsep == '/' and os.sep == '\\': + filenames = (f.replace(os.sep, os.altsep) for f in filenames) + filenames = filter_by_include_exclude(filenames, include, exclude) + return Classifier(filenames) -def _get_skips(environ): +def _get_skips(environ: MutableMapping[str, str]) -> set[str]: skips = environ.get('SKIP', '') return {skip.strip() for skip in skips.split(',') if skip.strip()} -def _hook_msg_start(hook, verbose): - return '{}{}'.format('[{}] '.format(hook.id) if verbose else '', hook.name) - - SKIPPED = 'Skipped' NO_FILES = '(no files to check)' -def _run_single_hook(classifier, hook, args, skips, cols): - filenames = classifier.filenames_for_hook(hook) +def _subtle_line(s: str, use_color: bool) -> None: + output.write_line(color.format_color(s, color.SUBTLE, use_color)) - if hook.language == 'pcre': - logger.warning( - '`{}` (from {}) uses the deprecated pcre language.\n' - 'The pcre language is scheduled for removal in pre-commit 2.x.\n' - 'The pygrep language is a more portable (and usually drop-in) ' - 'replacement.'.format(hook.id, hook.src), - ) + +def _run_single_hook( + classifier: Classifier, + hook: Hook, + skips: set[str], + cols: int, + diff_before: bytes, + verbose: bool, + use_color: bool, +) -> tuple[bool, bytes]: + filenames = tuple(classifier.filenames_for_hook(hook)) if hook.id in skips or hook.alias in skips: output.write( - get_hook_message( - _hook_msg_start(hook, args.verbose), + _full_msg( + start=hook.name, end_msg=SKIPPED, end_color=color.YELLOW, - use_color=args.color, + use_color=use_color, cols=cols, ), ) - return 0 + duration = None + retcode = 0 + diff_after = diff_before + files_modified = False + out = b'' elif not filenames and not hook.always_run: output.write( - get_hook_message( - _hook_msg_start(hook, args.verbose), + _full_msg( + start=hook.name, postfix=NO_FILES, end_msg=SKIPPED, end_color=color.TURQUOISE, - use_color=args.color, + use_color=use_color, cols=cols, ), ) - return 0 - - # Print the hook and the dots first in case the hook takes hella long to - # run. - output.write( - get_hook_message( - _hook_msg_start(hook, args.verbose), end_len=6, cols=cols, - ), - ) - sys.stdout.flush() - - diff_before = cmd_output( - 'git', 'diff', '--no-ext-diff', retcode=None, encoding=None, - ) - retcode, stdout, stderr = hook.run( - tuple(filenames) if hook.pass_filenames else (), - ) - diff_after = cmd_output( - 'git', 'diff', '--no-ext-diff', retcode=None, encoding=None, - ) - - file_modifications = diff_before != diff_after - - # If the hook makes changes, fail the commit - if file_modifications: - retcode = 1 - - if retcode: - retcode = 1 - print_color = color.RED - pass_fail = 'Failed' - else: + duration = None retcode = 0 - print_color = color.GREEN - pass_fail = 'Passed' + diff_after = diff_before + files_modified = False + out = b'' + else: + # print hook and dots first in case the hook takes a while to run + output.write(_start_msg(start=hook.name, end_len=6, cols=cols)) - output.write_line(color.format_color(pass_fail, print_color, args.color)) + if not hook.pass_filenames: + filenames = () + time_before = time.monotonic() + language = languages[hook.language] + with language.in_env(hook.prefix, hook.language_version): + retcode, out = language.run_hook( + hook.prefix, + hook.entry, + hook.args, + filenames, + is_local=hook.src == 'local', + require_serial=hook.require_serial, + color=use_color, + ) + duration = round(time.monotonic() - time_before, 2) or 0 + diff_after = _get_diff() - if ( - (stdout or stderr or file_modifications) and - (retcode or args.verbose or hook.verbose) - ): - output.write_line('hookid: {}\n'.format(hook.id)) + # if the hook makes changes, fail the commit + files_modified = diff_before != diff_after + + if retcode or files_modified: + print_color = color.RED + status = 'Failed' + else: + print_color = color.GREEN + status = 'Passed' + + output.write_line(color.format_color(status, print_color, use_color)) + + if verbose or hook.verbose or retcode or files_modified: + _subtle_line(f'- hook id: {hook.id}', use_color) + + if (verbose or hook.verbose) and duration is not None: + _subtle_line(f'- duration: {duration}s', use_color) + + if retcode: + _subtle_line(f'- exit code: {retcode}', use_color) # Print a message if failing due to file modifications - if file_modifications: - output.write('Files were modified by this hook.') - - if stdout or stderr: - output.write_line(' Additional output:') + if files_modified: + _subtle_line('- files were modified by this hook', use_color) + if out.strip(): + output.write_line() + output.write_line_b(out.strip(), logfile_name=hook.log_file) output.write_line() - for out in (stdout, stderr): - assert type(out) is bytes, type(out) - if out.strip(): - output.write_line(out.strip(), logfile_name=hook.log_file) - output.write_line() - - return retcode + return files_modified or bool(retcode), diff_after -def _compute_cols(hooks, verbose): +def _compute_cols(hooks: Sequence[Hook]) -> int: """Compute the number of columns to display hook messages. The widest that will be displayed is in the no files skipped case: Hook name...(no files to check) Skipped - - or in the verbose case - - Hook name [hookid]...(no files to check) Skipped """ if hooks: - name_len = max(len(_hook_msg_start(hook, verbose)) for hook in hooks) + name_len = max(_len_cjk(hook.name) for hook in hooks) else: name_len = 0 @@ -187,11 +250,17 @@ def _compute_cols(hooks, verbose): return max(cols, 80) -def _all_filenames(args): - if args.origin and args.source: - return git.get_changed_files(args.origin, args.source) +def _all_filenames(args: argparse.Namespace) -> Iterable[str]: + # these hooks do not operate on files + if args.hook_stage in { + 'post-checkout', 'post-commit', 'post-merge', 'post-rewrite', + 'pre-rebase', + }: + return () elif args.hook_stage in {'prepare-commit-msg', 'commit-msg'}: return (args.commit_msg_filename,) + elif args.from_ref and args.to_ref: + return git.get_changed_files(args.from_ref, args.to_ref) elif args.files: return args.files elif args.all_files: @@ -202,19 +271,37 @@ def _all_filenames(args): return git.get_staged_files() -def _run_hooks(config, hooks, args, environ): +def _get_diff() -> bytes: + _, out, _ = cmd_output_b( + 'git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules', + check=False, + ) + return out + + +def _run_hooks( + config: dict[str, Any], + hooks: Sequence[Hook], + skips: set[str], + args: argparse.Namespace, +) -> int: """Actually run the hooks.""" - skips = _get_skips(environ) - cols = _compute_cols(hooks, args.verbose) - filenames = _all_filenames(args) - filenames = filter_by_include_exclude(filenames, '', config['exclude']) - classifier = Classifier(filenames) + cols = _compute_cols(hooks) + classifier = Classifier.from_config( + _all_filenames(args), config['files'], config['exclude'], + ) retval = 0 + prior_diff = _get_diff() for hook in hooks: - retval |= _run_single_hook(classifier, hook, args, skips, cols) - if retval and config['fail_fast']: + current_retval, prior_diff = _run_single_hook( + classifier, hook, skips, cols, prior_diff, + verbose=args.verbose, use_color=args.color, + ) + retval |= current_retval + fail_fast = (config['fail_fast'] or hook.fail_fast or args.fail_fast) + if current_retval and fail_fast: break - if retval and args.show_diff_on_failure and git.has_diff(): + if retval and args.show_diff_on_failure and prior_diff: if args.all_files: output.write_line( 'pre-commit hook(s) made changes.\n' @@ -226,56 +313,113 @@ def _run_hooks(config, hooks, args, environ): output.write_line('All changes made by hooks:') # args.color is a boolean. # See user_color function in color.py + git_color_opt = 'always' if args.color else 'never' subprocess.call(( 'git', '--no-pager', 'diff', '--no-ext-diff', - '--color={}'.format({True: 'always', False: 'never'}[args.color]), + f'--color={git_color_opt}', )) return retval -def _has_unmerged_paths(): - _, stdout, _ = cmd_output('git', 'ls-files', '--unmerged') +def _has_unmerged_paths() -> bool: + _, stdout, _ = cmd_output_b('git', 'ls-files', '--unmerged') return bool(stdout.strip()) -def _has_unstaged_config(config_file): - retcode, _, _ = cmd_output( - 'git', 'diff', '--no-ext-diff', '--exit-code', config_file, - retcode=None, +def _has_unstaged_config(config_file: str) -> bool: + retcode, _, _ = cmd_output_b( + 'git', 'diff', '--quiet', '--no-ext-diff', config_file, check=False, ) # be explicit, other git errors don't mean it has an unstaged config. return retcode == 1 -def run(config_file, store, args, environ=os.environ): - no_stash = args.all_files or bool(args.files) +def run( + config_file: str, + store: Store, + args: argparse.Namespace, + environ: MutableMapping[str, str] = os.environ, +) -> int: + stash = not args.all_files and not args.files # Check if we have unresolved merge conflict files and fail fast. - if _has_unmerged_paths(): + if stash and _has_unmerged_paths(): logger.error('Unmerged files. Resolve before committing.') return 1 - if bool(args.source) != bool(args.origin): - logger.error('Specify both --origin and --source.') + if bool(args.from_ref) != bool(args.to_ref): + logger.error('Specify both --from-ref and --to-ref.') return 1 - if _has_unstaged_config(config_file) and not no_stash: + if stash and _has_unstaged_config(config_file): logger.error( - 'Your pre-commit configuration is unstaged.\n' - '`git add {}` to fix this.'.format(config_file), + f'Your pre-commit configuration is unstaged.\n' + f'`git add {config_file}` to fix this.', ) return 1 + if ( + args.hook_stage in {'prepare-commit-msg', 'commit-msg'} and + not args.commit_msg_filename + ): + logger.error( + f'`--commit-msg-filename` is required for ' + f'`--hook-stage {args.hook_stage}`', + ) + return 1 + # prevent recursive post-checkout hooks (#1418) + if ( + args.hook_stage == 'post-checkout' and + environ.get('_PRE_COMMIT_SKIP_POST_CHECKOUT') + ): + return 0 - # Expose origin / source as environment variables for hooks to consume - if args.origin and args.source: - environ['PRE_COMMIT_ORIGIN'] = args.origin - environ['PRE_COMMIT_SOURCE'] = args.source + # Expose prepare_commit_message_source / commit_object_name + # as environment variables for the hooks + if args.prepare_commit_message_source: + environ['PRE_COMMIT_COMMIT_MSG_SOURCE'] = ( + args.prepare_commit_message_source + ) - if no_stash: - ctx = noop_context() - else: - ctx = staged_files_only(store.directory) + if args.commit_object_name: + environ['PRE_COMMIT_COMMIT_OBJECT_NAME'] = args.commit_object_name + + # Expose from-ref / to-ref as environment variables for hooks to consume + if args.from_ref and args.to_ref: + # legacy names + environ['PRE_COMMIT_ORIGIN'] = args.from_ref + environ['PRE_COMMIT_SOURCE'] = args.to_ref + # new names + environ['PRE_COMMIT_FROM_REF'] = args.from_ref + environ['PRE_COMMIT_TO_REF'] = args.to_ref + + if args.pre_rebase_upstream and args.pre_rebase_branch: + environ['PRE_COMMIT_PRE_REBASE_UPSTREAM'] = args.pre_rebase_upstream + environ['PRE_COMMIT_PRE_REBASE_BRANCH'] = args.pre_rebase_branch + + if ( + args.remote_name and args.remote_url and + args.remote_branch and args.local_branch + ): + environ['PRE_COMMIT_LOCAL_BRANCH'] = args.local_branch + environ['PRE_COMMIT_REMOTE_BRANCH'] = args.remote_branch + environ['PRE_COMMIT_REMOTE_NAME'] = args.remote_name + environ['PRE_COMMIT_REMOTE_URL'] = args.remote_url + + if args.checkout_type: + environ['PRE_COMMIT_CHECKOUT_TYPE'] = args.checkout_type + + if args.is_squash_merge: + environ['PRE_COMMIT_IS_SQUASH_MERGE'] = args.is_squash_merge + + if args.rewrite_command: + environ['PRE_COMMIT_REWRITE_COMMAND'] = args.rewrite_command + + # Set pre_commit flag + environ['PRE_COMMIT'] = '1' + + with contextlib.ExitStack() as exit_stack: + if stash: + exit_stack.enter_context(staged_files_only(store.directory)) - with ctx: config = load_config(config_file) hooks = [ hook @@ -286,12 +430,19 @@ def run(config_file, store, args, environ=os.environ): if args.hook and not hooks: output.write_line( - 'No hook with id `{}` in stage `{}`'.format( - args.hook, args.hook_stage, - ), + f'No hook with id `{args.hook}` in stage `{args.hook_stage}`', ) return 1 - install_hook_envs(hooks, store) + skips = _get_skips(environ) + to_install = [ + hook + for hook in hooks + if hook.id not in skips and hook.alias not in skips + ] + install_hook_envs(to_install, store) - return _run_hooks(config, hooks, args, environ) + return _run_hooks(config, hooks, skips, args) + + # https://github.com/python/mypy/issues/7726 + raise AssertionError('unreachable') diff --git a/pre_commit/commands/sample_config.py b/pre_commit/commands/sample_config.py index 38320f67..ce22f65e 100644 --- a/pre_commit/commands/sample_config.py +++ b/pre_commit/commands/sample_config.py @@ -1,18 +1,10 @@ -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - - -# TODO: maybe `git ls-remote git://github.com/pre-commit/pre-commit-hooks` to -# determine the latest revision? This adds ~200ms from my tests (and is -# significantly faster than https:// or http://). For now, periodically -# manually updating the revision is fine. +from __future__ import annotations SAMPLE_CONFIG = '''\ # See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v2.0.0 + rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer @@ -21,6 +13,6 @@ repos: ''' -def sample_config(): +def sample_config() -> int: print(SAMPLE_CONFIG, end='') return 0 diff --git a/pre_commit/commands/try_repo.py b/pre_commit/commands/try_repo.py index 3e256ad8..539ed3c2 100644 --- a/pre_commit/commands/try_repo.py +++ b/pre_commit/commands/try_repo.py @@ -1,11 +1,9 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import collections +import argparse import logging import os.path - -from aspy.yaml import ordered_dump +import tempfile import pre_commit.constants as C from pre_commit import git @@ -13,16 +11,16 @@ from pre_commit import output from pre_commit.clientlib import load_manifest from pre_commit.commands.run import run from pre_commit.store import Store -from pre_commit.util import cmd_output -from pre_commit.util import tmpdir +from pre_commit.util import cmd_output_b from pre_commit.xargs import xargs +from pre_commit.yaml import yaml_dump logger = logging.getLogger(__name__) -def _repo_ref(tmpdir, repo, ref): +def _repo_ref(tmpdir: str, repo: str, ref: str | None) -> tuple[str, str]: # if `ref` is explicitly passed, use it - if ref: + if ref is not None: return repo, ref ref = git.head_rev(repo) @@ -31,8 +29,8 @@ def _repo_ref(tmpdir, repo, ref): logger.warning('Creating temporary repo with uncommitted changes...') shadow = os.path.join(tmpdir, 'shadow-repo') - cmd_output('git', 'clone', repo, shadow) - cmd_output('git', 'checkout', ref, '-b', '_pc_tmp', cwd=shadow) + cmd_output_b('git', 'clone', repo, shadow) + cmd_output_b('git', 'checkout', ref, '-b', '_pc_tmp', cwd=shadow) idx = git.git_path('index', repo=shadow) objs = git.git_path('objects', repo=shadow) @@ -42,7 +40,7 @@ def _repo_ref(tmpdir, repo, ref): if staged_files: xargs(('git', 'add', '--'), staged_files, cwd=repo, env=env) - cmd_output('git', 'add', '-u', cwd=repo, env=env) + cmd_output_b('git', 'add', '-u', cwd=repo, env=env) git.commit(repo=shadow) return shadow, git.head_rev(shadow) @@ -50,8 +48,8 @@ def _repo_ref(tmpdir, repo, ref): return repo, ref -def try_repo(args): - with tmpdir() as tempdir: +def try_repo(args: argparse.Namespace) -> int: + with tempfile.TemporaryDirectory() as tempdir: repo, ref = _repo_ref(tempdir, args.repo, args.ref) store = Store(tempdir) @@ -63,9 +61,8 @@ def try_repo(args): manifest = sorted(manifest, key=lambda hook: hook['id']) hooks = [{'id': hook['id']} for hook in manifest] - items = (('repo', repo), ('rev', ref), ('hooks', hooks)) - config = {'repos': [collections.OrderedDict(items)]} - config_s = ordered_dump(config, **C.YAML_DUMP_KWARGS) + config = {'repos': [{'repo': repo, 'rev': ref, 'hooks': hooks}]} + config_s = yaml_dump(config) config_filename = os.path.join(tempdir, C.CONFIG_FILE) with open(config_filename, 'w') as cfg: diff --git a/pre_commit/commands/validate_config.py b/pre_commit/commands/validate_config.py new file mode 100644 index 00000000..b3de635b --- /dev/null +++ b/pre_commit/commands/validate_config.py @@ -0,0 +1,18 @@ +from __future__ import annotations + +from collections.abc import Sequence + +from pre_commit import clientlib + + +def validate_config(filenames: Sequence[str]) -> int: + ret = 0 + + for filename in filenames: + try: + clientlib.load_config(filename) + except clientlib.InvalidConfigError as e: + print(e) + ret = 1 + + return ret diff --git a/pre_commit/commands/validate_manifest.py b/pre_commit/commands/validate_manifest.py new file mode 100644 index 00000000..8493c6e1 --- /dev/null +++ b/pre_commit/commands/validate_manifest.py @@ -0,0 +1,18 @@ +from __future__ import annotations + +from collections.abc import Sequence + +from pre_commit import clientlib + + +def validate_manifest(filenames: Sequence[str]) -> int: + ret = 0 + + for filename in filenames: + try: + clientlib.load_manifest(filename) + except clientlib.InvalidManifestError as e: + print(e) + ret = 1 + + return ret diff --git a/pre_commit/constants.py b/pre_commit/constants.py index 307b09a4..79a9bb69 100644 --- a/pre_commit/constants.py +++ b/pre_commit/constants.py @@ -1,26 +1,13 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import importlib_metadata # TODO: importlib.metadata py38? +import importlib.metadata CONFIG_FILE = '.pre-commit-config.yaml' MANIFEST_FILE = '.pre-commit-hooks.yaml' -YAML_DUMP_KWARGS = { - 'default_flow_style': False, - # Use unicode - 'encoding': None, - 'indent': 4, -} - -# Bump when installation changes in a backwards / forwards incompatible way -INSTALLED_STATE_VERSION = '1' # Bump when modifying `empty_template` LOCAL_REPO_VERSION = '1' -VERSION = importlib_metadata.version('pre_commit') - -# `manual` is not invoked by any installed git hook. See #719 -STAGES = ('commit', 'prepare-commit-msg', 'commit-msg', 'manual', 'push') +VERSION = importlib.metadata.version('pre_commit') DEFAULT = 'default' diff --git a/pre_commit/envcontext.py b/pre_commit/envcontext.py index 82538df2..d4d24118 100644 --- a/pre_commit/envcontext.py +++ b/pre_commit/envcontext.py @@ -1,19 +1,28 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import collections import contextlib +import enum import os +from collections.abc import Generator +from collections.abc import MutableMapping +from typing import NamedTuple +from typing import Union + +_Unset = enum.Enum('_Unset', 'UNSET') +UNSET = _Unset.UNSET -UNSET = collections.namedtuple('UNSET', ())() +class Var(NamedTuple): + name: str + default: str = '' -Var = collections.namedtuple('Var', ('name', 'default')) -Var.__new__.__defaults__ = ('',) +SubstitutionT = tuple[Union[str, Var], ...] +ValueT = Union[str, _Unset, SubstitutionT] +PatchesT = tuple[tuple[str, ValueT], ...] -def format_env(parts, env): +def format_env(parts: SubstitutionT, env: MutableMapping[str, str]) -> str: return ''.join( env.get(part.name, part.default) if isinstance(part, Var) else part for part in parts @@ -21,7 +30,10 @@ def format_env(parts, env): @contextlib.contextmanager -def envcontext(patch, _env=None): +def envcontext( + patch: PatchesT, + _env: MutableMapping[str, str] | None = None, +) -> Generator[None]: """In this context, `os.environ` is modified according to `patch`. `patch` is an iterable of 2-tuples (key, value): @@ -33,7 +45,7 @@ def envcontext(patch, _env=None): replaced with the previous environment """ env = os.environ if _env is None else _env - before = env.copy() + before = dict(env) for k, v in patch: if v is UNSET: diff --git a/pre_commit/error_handler.py b/pre_commit/error_handler.py index 946f134c..4f0e0573 100644 --- a/pre_commit/error_handler.py +++ b/pre_commit/error_handler.py @@ -1,54 +1,81 @@ -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals +from __future__ import annotations import contextlib +import functools import os.path +import sys import traceback +from collections.abc import Generator +from typing import IO -import six - -from pre_commit import five +import pre_commit.constants as C from pre_commit import output +from pre_commit.errors import FatalError from pre_commit.store import Store +from pre_commit.util import cmd_output_b +from pre_commit.util import force_bytes -class FatalError(RuntimeError): - pass +def _log_and_exit( + msg: str, + ret_code: int, + exc: BaseException, + formatted: str, +) -> None: + error_msg = f'{msg}: {type(exc).__name__}: '.encode() + force_bytes(exc) + output.write_line_b(error_msg) + _, git_version_b, _ = cmd_output_b('git', '--version', check=False) + git_version = git_version_b.decode(errors='backslashreplace').rstrip() -def _to_bytes(exc): - try: - return bytes(exc) - except Exception: - return six.text_type(exc).encode('UTF-8') + storedir = Store().directory + log_path = os.path.join(storedir, 'pre-commit.log') + with contextlib.ExitStack() as ctx: + if os.access(storedir, os.W_OK): + output.write_line(f'Check the log at {log_path}') + log: IO[bytes] = ctx.enter_context(open(log_path, 'wb')) + else: # pragma: win32 no cover + output.write_line(f'Failed to write to log at {log_path}') + log = sys.stdout.buffer + _log_line = functools.partial(output.write_line, stream=log) + _log_line_b = functools.partial(output.write_line_b, stream=log) -def _log_and_exit(msg, exc, formatted): - error_msg = b''.join(( - five.to_bytes(msg), b': ', - five.to_bytes(type(exc).__name__), b': ', - _to_bytes(exc), b'\n', - )) - output.write(error_msg) - store = Store() - log_path = os.path.join(store.directory, 'pre-commit.log') - output.write_line('Check the log at {}'.format(log_path)) - with open(log_path, 'wb') as log: - output.write(error_msg, stream=log) - output.write_line(formatted, stream=log) - raise SystemExit(1) + _log_line('### version information') + _log_line() + _log_line('```') + _log_line(f'pre-commit version: {C.VERSION}') + _log_line(f'git --version: {git_version}') + _log_line('sys.version:') + for line in sys.version.splitlines(): + _log_line(f' {line}') + _log_line(f'sys.executable: {sys.executable}') + _log_line(f'os.name: {os.name}') + _log_line(f'sys.platform: {sys.platform}') + _log_line('```') + _log_line() + + _log_line('### error information') + _log_line() + _log_line('```') + _log_line_b(error_msg) + _log_line('```') + _log_line() + _log_line('```') + _log_line(formatted.rstrip()) + _log_line('```') + raise SystemExit(ret_code) @contextlib.contextmanager -def error_handler(): +def error_handler() -> Generator[None]: try: yield except (Exception, KeyboardInterrupt) as e: if isinstance(e, FatalError): - msg = 'An error has occurred' + msg, ret_code = 'An error has occurred', 1 elif isinstance(e, KeyboardInterrupt): - msg = 'Interrupted (^C)' + msg, ret_code = 'Interrupted (^C)', 130 else: - msg = 'An unexpected error has occurred' - _log_and_exit(msg, e, traceback.format_exc()) + msg, ret_code = 'An unexpected error has occurred', 3 + _log_and_exit(msg, ret_code, e, traceback.format_exc()) diff --git a/pre_commit/errors.py b/pre_commit/errors.py new file mode 100644 index 00000000..eac34faa --- /dev/null +++ b/pre_commit/errors.py @@ -0,0 +1,5 @@ +from __future__ import annotations + + +class FatalError(RuntimeError): + pass diff --git a/pre_commit/file_lock.py b/pre_commit/file_lock.py index cf9aeac5..6223f869 100644 --- a/pre_commit/file_lock.py +++ b/pre_commit/file_lock.py @@ -1,11 +1,13 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import contextlib import errno +import sys +from collections.abc import Callable +from collections.abc import Generator -try: # pragma: no cover (windows) +if sys.platform == 'win32': # pragma: no cover (windows) import msvcrt # https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/locking @@ -15,15 +17,18 @@ try: # pragma: no cover (windows) _region = 0xffff @contextlib.contextmanager - def _locked(fileno, blocked_cb): + def _locked( + fileno: int, + blocked_cb: Callable[[], None], + ) -> Generator[None]: try: msvcrt.locking(fileno, msvcrt.LK_NBLCK, _region) - except IOError: + except OSError: blocked_cb() while True: try: msvcrt.locking(fileno, msvcrt.LK_LOCK, _region) - except IOError as e: + except OSError as e: # Locking violation. Returned when the _LK_LOCK or _LK_RLCK # flag is specified and the file cannot be locked after 10 # attempts. @@ -41,14 +46,17 @@ try: # pragma: no cover (windows) # "Regions should be locked only briefly and should be unlocked # before closing a file or exiting the program." msvcrt.locking(fileno, msvcrt.LK_UNLCK, _region) -except ImportError: # pragma: windows no cover +else: # pragma: win32 no cover import fcntl @contextlib.contextmanager - def _locked(fileno, blocked_cb): + def _locked( + fileno: int, + blocked_cb: Callable[[], None], + ) -> Generator[None]: try: fcntl.flock(fileno, fcntl.LOCK_EX | fcntl.LOCK_NB) - except IOError: # pragma: no cover (tests are single-threaded) + except OSError: # pragma: no cover (tests are single-threaded) blocked_cb() fcntl.flock(fileno, fcntl.LOCK_EX) try: @@ -58,7 +66,10 @@ except ImportError: # pragma: windows no cover @contextlib.contextmanager -def lock(path, blocked_cb): +def lock( + path: str, + blocked_cb: Callable[[], None], +) -> Generator[None]: with open(path, 'a+') as f: with _locked(f.fileno(), blocked_cb): yield diff --git a/pre_commit/five.py b/pre_commit/five.py deleted file mode 100644 index 3b94a927..00000000 --- a/pre_commit/five.py +++ /dev/null @@ -1,15 +0,0 @@ -from __future__ import absolute_import -from __future__ import unicode_literals - -import six - - -def to_text(s): - return s if isinstance(s, six.text_type) else s.decode('UTF-8') - - -def to_bytes(s): - return s if isinstance(s, bytes) else s.encode('UTF-8') - - -n = to_bytes if six.PY2 else to_text diff --git a/pre_commit/git.py b/pre_commit/git.py index c51930e7..ec1928f3 100644 --- a/pre_commit/git.py +++ b/pre_commit/git.py @@ -1,16 +1,22 @@ -from __future__ import unicode_literals +from __future__ import annotations import logging import os.path import sys +from collections.abc import Mapping +from pre_commit.errors import FatalError +from pre_commit.util import CalledProcessError from pre_commit.util import cmd_output - +from pre_commit.util import cmd_output_b logger = logging.getLogger(__name__) +# see #2046 +NO_FS_MONITOR = ('-c', 'core.useBuiltinFSMonitor=false') -def zsplit(s): + +def zsplit(s: str) -> list[str]: s = s.strip('\0') if s: return s.split('\0') @@ -18,7 +24,7 @@ def zsplit(s): return [] -def no_git_env(_env=None): +def no_git_env(_env: Mapping[str, str] | None = None) -> dict[str, str]: # Too many bugs dealing with environment variables and GIT: # https://github.com/pre-commit/pre-commit/issues/300 # In git 2.6.3 (maybe others), git exports GIT_WORK_TREE while running @@ -31,30 +37,63 @@ def no_git_env(_env=None): return { k: v for k, v in _env.items() if not k.startswith('GIT_') or - k in {'GIT_EXEC_PATH', 'GIT_SSH', 'GIT_SSH_COMMAND'} + k.startswith(('GIT_CONFIG_KEY_', 'GIT_CONFIG_VALUE_')) or + k in { + 'GIT_EXEC_PATH', 'GIT_SSH', 'GIT_SSH_COMMAND', 'GIT_SSL_CAINFO', + 'GIT_SSL_NO_VERIFY', 'GIT_CONFIG_COUNT', + 'GIT_HTTP_PROXY_AUTHMETHOD', + 'GIT_ALLOW_PROTOCOL', + 'GIT_ASKPASS', + } } -def get_root(): - return cmd_output('git', 'rev-parse', '--show-toplevel')[1].strip() +def get_root() -> str: + # Git 2.25 introduced a change to "rev-parse --show-toplevel" that exposed + # underlying volumes for Windows drives mapped with SUBST. We use + # "rev-parse --show-cdup" to get the appropriate path, but must perform + # an extra check to see if we are in the .git directory. + try: + root = os.path.abspath( + cmd_output('git', 'rev-parse', '--show-cdup')[1].strip(), + ) + inside_git_dir = cmd_output( + 'git', 'rev-parse', '--is-inside-git-dir', + )[1].strip() + except CalledProcessError: + raise FatalError( + 'git failed. Is it installed, and are you in a Git repository ' + 'directory?', + ) + if inside_git_dir != 'false': + raise FatalError( + 'git toplevel unexpectedly empty! make sure you are not ' + 'inside the `.git` directory of your repository.', + ) + return root -def get_git_dir(git_root='.'): - opts = ('--git-common-dir', '--git-dir') - _, out, _ = cmd_output('git', 'rev-parse', *opts, cwd=git_root) - for line, opt in zip(out.splitlines(), opts): - if line != opt: # pragma: no branch (git < 2.5) - return os.path.normpath(os.path.join(git_root, line)) +def get_git_dir(git_root: str = '.') -> str: + opt = '--git-dir' + _, out, _ = cmd_output('git', 'rev-parse', opt, cwd=git_root) + git_dir = out.strip() + if git_dir != opt: + return os.path.normpath(os.path.join(git_root, git_dir)) else: raise AssertionError('unreachable: no git dir') -def get_remote_url(git_root): - ret = cmd_output('git', 'config', 'remote.origin.url', cwd=git_root)[1] - return ret.strip() +def get_git_common_dir(git_root: str = '.') -> str: + opt = '--git-common-dir' + _, out, _ = cmd_output('git', 'rev-parse', opt, cwd=git_root) + git_common_dir = out.strip() + if git_common_dir != opt: + return os.path.normpath(os.path.join(git_root, git_common_dir)) + else: # pragma: no cover (git < 2.5) + return get_git_dir(git_root) -def is_in_merge_conflict(): +def is_in_merge_conflict() -> bool: git_dir = get_git_dir('.') return ( os.path.exists(os.path.join(git_dir, 'MERGE_MSG')) and @@ -62,17 +101,17 @@ def is_in_merge_conflict(): ) -def parse_merge_msg_for_conflicts(merge_msg): +def parse_merge_msg_for_conflicts(merge_msg: bytes) -> list[str]: # Conflicted files start with tabs return [ - line.lstrip(b'#').strip().decode('UTF-8') + line.lstrip(b'#').strip().decode() for line in merge_msg.splitlines() # '#\t' for git 2.4.1 if line.startswith((b'\t', b'#\t')) ] -def get_conflicted_files(): +def get_conflicted_files() -> set[str]: logger.info('Checking merge-conflict files only.') # Need to get the conflicted files from the MERGE_MSG because they could # have resolved the conflict by choosing one side or the other @@ -87,13 +126,13 @@ def get_conflicted_files(): merge_diff_filenames = zsplit( cmd_output( 'git', 'diff', '--name-only', '--no-ext-diff', '-z', - '-m', tree_hash, 'HEAD', 'MERGE_HEAD', + '-m', tree_hash, 'HEAD', 'MERGE_HEAD', '--', )[1], ) return set(merge_conflict_filenames) | set(merge_diff_filenames) -def get_staged_files(cwd=None): +def get_staged_files(cwd: str | None = None) -> list[str]: return zsplit( cmd_output( 'git', 'diff', '--staged', '--name-only', '--no-ext-diff', '-z', @@ -104,85 +143,103 @@ def get_staged_files(cwd=None): ) -def intent_to_add_files(): - _, stdout_binary, _ = cmd_output('git', 'status', '--porcelain', '-z') - parts = list(reversed(zsplit(stdout_binary))) - intent_to_add = [] - while parts: - line = parts.pop() - status, filename = line[:3], line[3:] - if status[0] in {'C', 'R'}: # renames / moves have an additional arg - parts.pop() - if status[1] == 'A': - intent_to_add.append(filename) - return intent_to_add +def intent_to_add_files() -> list[str]: + _, stdout, _ = cmd_output( + 'git', 'diff', '--no-ext-diff', '--ignore-submodules', + '--diff-filter=A', '--name-only', '-z', + ) + return zsplit(stdout) -def get_all_files(): +def get_all_files() -> list[str]: return zsplit(cmd_output('git', 'ls-files', '-z')[1]) -def get_changed_files(new, old): - return zsplit( - cmd_output( - 'git', 'diff', '--name-only', '--no-ext-diff', '-z', - '{}...{}'.format(old, new), - )[1], - ) +def get_changed_files(old: str, new: str) -> list[str]: + diff_cmd = ('git', 'diff', '--name-only', '--no-ext-diff', '-z') + try: + _, out, _ = cmd_output(*diff_cmd, f'{old}...{new}') + except CalledProcessError: # pragma: no cover (new git) + # on newer git where old and new do not have a merge base git fails + # so we try a full diff (this is what old git did for us!) + _, out, _ = cmd_output(*diff_cmd, f'{old}..{new}') + + return zsplit(out) -def head_rev(remote): +def head_rev(remote: str) -> str: _, out, _ = cmd_output('git', 'ls-remote', '--exit-code', remote, 'HEAD') return out.split()[0] -def has_diff(*args, **kwargs): - repo = kwargs.pop('repo', '.') - assert not kwargs, kwargs - cmd = ('git', 'diff', '--quiet', '--no-ext-diff') + args - return cmd_output(*cmd, cwd=repo, retcode=None)[0] +def has_diff(*args: str, repo: str = '.') -> bool: + cmd = ('git', 'diff', '--quiet', '--no-ext-diff', *args) + return cmd_output_b(*cmd, cwd=repo, check=False)[0] == 1 -def init_repo(path, remote): +def has_core_hookpaths_set() -> bool: + _, out, _ = cmd_output_b('git', 'config', 'core.hooksPath', check=False) + return bool(out.strip()) + + +def init_repo(path: str, remote: str) -> None: if os.path.isdir(remote): remote = os.path.abspath(remote) + git = ('git', *NO_FS_MONITOR) env = no_git_env() - cmd_output('git', 'init', path, env=env) - cmd_output('git', 'remote', 'add', 'origin', remote, cwd=path, env=env) + # avoid the user's template so that hooks do not recurse + cmd_output_b(*git, 'init', '--template=', path, env=env) + cmd_output_b(*git, 'remote', 'add', 'origin', remote, cwd=path, env=env) -def commit(repo='.'): +def commit(repo: str = '.') -> None: env = no_git_env() name, email = 'pre-commit', 'asottile+pre-commit@umich.edu' env['GIT_AUTHOR_NAME'] = env['GIT_COMMITTER_NAME'] = name env['GIT_AUTHOR_EMAIL'] = env['GIT_COMMITTER_EMAIL'] = email cmd = ('git', 'commit', '--no-edit', '--no-gpg-sign', '-n', '-minit') - cmd_output(*cmd, cwd=repo, env=env) + cmd_output_b(*cmd, cwd=repo, env=env) -def git_path(name, repo='.'): +def git_path(name: str, repo: str = '.') -> str: _, out, _ = cmd_output('git', 'rev-parse', '--git-path', name, cwd=repo) return os.path.join(repo, out.strip()) -def check_for_cygwin_mismatch(): +def check_for_cygwin_mismatch() -> None: """See https://github.com/pre-commit/pre-commit/issues/354""" if sys.platform in ('cygwin', 'win32'): # pragma: no cover (windows) is_cygwin_python = sys.platform == 'cygwin' - toplevel = cmd_output('git', 'rev-parse', '--show-toplevel')[1] + try: + toplevel = get_root() + except FatalError: # skip the check if we're not in a git repo + return is_cygwin_git = toplevel.startswith('/') if is_cygwin_python ^ is_cygwin_git: exe_type = {True: '(cygwin)', False: '(windows)'} - logger.warn( - 'pre-commit has detected a mix of cygwin python / git\n' - 'This combination is not supported, it is likely you will ' - 'receive an error later in the program.\n' - 'Make sure to use cygwin git+python while using cygwin\n' - 'These can be installed through the cygwin installer.\n' - ' - python {}\n' - ' - git {}\n'.format( - exe_type[is_cygwin_python], exe_type[is_cygwin_git], - ), + logger.warning( + f'pre-commit has detected a mix of cygwin python / git\n' + f'This combination is not supported, it is likely you will ' + f'receive an error later in the program.\n' + f'Make sure to use cygwin git+python while using cygwin\n' + f'These can be installed through the cygwin installer.\n' + f' - python {exe_type[is_cygwin_python]}\n' + f' - git {exe_type[is_cygwin_git]}\n', ) + + +def get_best_candidate_tag(rev: str, git_repo: str) -> str: + """Get the best tag candidate. + + Multiple tags can exist on a SHA. Sometimes a moving tag is attached + to a version tag. Try to pick the tag that looks like a version. + """ + tags = cmd_output( + 'git', *NO_FS_MONITOR, 'tag', '--points-at', rev, cwd=git_repo, + )[1].splitlines() + for tag in tags: + if '.' in tag: + return tag + return rev diff --git a/pre_commit/hook.py b/pre_commit/hook.py new file mode 100644 index 00000000..309cd5be --- /dev/null +++ b/pre_commit/hook.py @@ -0,0 +1,60 @@ +from __future__ import annotations + +import logging +from collections.abc import Sequence +from typing import Any +from typing import NamedTuple + +from pre_commit.prefix import Prefix + +logger = logging.getLogger('pre_commit') + + +class Hook(NamedTuple): + src: str + prefix: Prefix + id: str + name: str + entry: str + language: str + alias: str + files: str + exclude: str + types: Sequence[str] + types_or: Sequence[str] + exclude_types: Sequence[str] + additional_dependencies: Sequence[str] + args: Sequence[str] + always_run: bool + fail_fast: bool + pass_filenames: bool + description: str + language_version: str + log_file: str + minimum_pre_commit_version: str + require_serial: bool + stages: Sequence[str] + verbose: bool + + @property + def install_key(self) -> tuple[Prefix, str, str, tuple[str, ...]]: + return ( + self.prefix, + self.language, + self.language_version, + tuple(self.additional_dependencies), + ) + + @classmethod + def create(cls, src: str, prefix: Prefix, dct: dict[str, Any]) -> Hook: + # TODO: have cfgv do this (?) + extra_keys = set(dct) - _KEYS + if extra_keys: + logger.warning( + f'Unexpected key(s) present on {src} => {dct["id"]}: ' + f'{", ".join(sorted(extra_keys))}', + ) + return cls(src=src, prefix=prefix, **{k: dct[k] for k in _KEYS}) + + +_KEYS = frozenset(set(Hook._fields) - {'src', 'prefix'}) diff --git a/pre_commit/lang_base.py b/pre_commit/lang_base.py new file mode 100644 index 00000000..198e9365 --- /dev/null +++ b/pre_commit/lang_base.py @@ -0,0 +1,196 @@ +from __future__ import annotations + +import contextlib +import os +import random +import re +import shlex +import sys +from collections.abc import Generator +from collections.abc import Sequence +from typing import Any +from typing import ContextManager +from typing import NoReturn +from typing import Protocol + +import pre_commit.constants as C +from pre_commit import parse_shebang +from pre_commit import xargs +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output_b + +FIXED_RANDOM_SEED = 1542676187 + +SHIMS_RE = re.compile(r'[/\\]shims[/\\]') + + +class Language(Protocol): + # Use `None` for no installation / environment + @property + def ENVIRONMENT_DIR(self) -> str | None: ... + # return a value to replace `'default` for `language_version` + def get_default_version(self) -> str: ... + # return whether the environment is healthy (or should be rebuilt) + def health_check(self, prefix: Prefix, version: str) -> str | None: ... + + # install a repository for the given language and language_version + def install_environment( + self, + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], + ) -> None: + ... + + # modify the environment for hook execution + def in_env(self, prefix: Prefix, version: str) -> ContextManager[None]: ... + + # execute a hook and return the exit code and output + def run_hook( + self, + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, + ) -> tuple[int, bytes]: + ... + + +def exe_exists(exe: str) -> bool: + found = parse_shebang.find_executable(exe) + if found is None: # exe exists + return False + + homedir = os.path.expanduser('~') + try: + common: str | None = os.path.commonpath((found, homedir)) + except ValueError: # on windows, different drives raises ValueError + common = None + + return ( + # it is not in a /shims/ directory + not SHIMS_RE.search(found) and + ( + # the homedir is / (docker, service user, etc.) + os.path.dirname(homedir) == homedir or + # the exe is not contained in the home directory + common != homedir + ) + ) + + +def setup_cmd(prefix: Prefix, cmd: tuple[str, ...], **kwargs: Any) -> None: + cmd_output_b(*cmd, cwd=prefix.prefix_dir, **kwargs) + + +def environment_dir(prefix: Prefix, d: str, language_version: str) -> str: + return prefix.path(f'{d}-{language_version}') + + +def assert_version_default(binary: str, version: str) -> None: + if version != C.DEFAULT: + raise AssertionError( + f'for now, pre-commit requires system-installed {binary} -- ' + f'you selected `language_version: {version}`', + ) + + +def assert_no_additional_deps( + lang: str, + additional_deps: Sequence[str], +) -> None: + if additional_deps: + raise AssertionError( + f'for now, pre-commit does not support ' + f'additional_dependencies for {lang} -- ' + f'you selected `additional_dependencies: {additional_deps}`', + ) + + +def basic_get_default_version() -> str: + return C.DEFAULT + + +def basic_health_check(prefix: Prefix, language_version: str) -> str | None: + return None + + +def no_install( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> NoReturn: + raise AssertionError('This language is not installable') + + +@contextlib.contextmanager +def no_env(prefix: Prefix, version: str) -> Generator[None]: + yield + + +def target_concurrency() -> int: + if 'PRE_COMMIT_NO_CONCURRENCY' in os.environ: + return 1 + else: + # Travis appears to have a bunch of CPUs, but we can't use them all. + if 'TRAVIS' in os.environ: + return 2 + else: + return xargs.cpu_count() + + +def _shuffled(seq: Sequence[str]) -> list[str]: + """Deterministically shuffle""" + fixed_random = random.Random() + fixed_random.seed(FIXED_RANDOM_SEED, version=1) + + seq = list(seq) + fixed_random.shuffle(seq) + return seq + + +def run_xargs( + cmd: tuple[str, ...], + file_args: Sequence[str], + *, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + if require_serial: + jobs = 1 + else: + # Shuffle the files so that they more evenly fill out the xargs + # partitions, but do it deterministically in case a hook cares about + # ordering. + file_args = _shuffled(file_args) + jobs = target_concurrency() + return xargs.xargs(cmd, file_args, target_concurrency=jobs, color=color) + + +def hook_cmd(entry: str, args: Sequence[str]) -> tuple[str, ...]: + cmd = shlex.split(entry) + if cmd[:2] == ['pre-commit', 'hazmat']: + cmd = [sys.executable, '-m', 'pre_commit.commands.hazmat', *cmd[2:]] + return (*cmd, *args) + + +def basic_run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + return run_xargs( + hook_cmd(entry, args), + file_args, + require_serial=require_serial, + color=color, + ) diff --git a/pre_commit/languages/all.py b/pre_commit/languages/all.py deleted file mode 100644 index 6d85ddf1..00000000 --- a/pre_commit/languages/all.py +++ /dev/null @@ -1,69 +0,0 @@ -from __future__ import unicode_literals - -from pre_commit.languages import docker -from pre_commit.languages import docker_image -from pre_commit.languages import fail -from pre_commit.languages import golang -from pre_commit.languages import node -from pre_commit.languages import pcre -from pre_commit.languages import pygrep -from pre_commit.languages import python -from pre_commit.languages import python_venv -from pre_commit.languages import ruby -from pre_commit.languages import rust -from pre_commit.languages import script -from pre_commit.languages import swift -from pre_commit.languages import system - -# A language implements the following constant and functions in its module: -# -# # Use None for no environment -# ENVIRONMENT_DIR = 'foo_env' -# -# def get_default_version(): -# """Return a value to replace the 'default' value for language_version. -# -# return 'default' if there is no better option. -# """ -# -# def healthy(prefix, language_version): -# """Return whether or not the environment is considered functional.""" -# -# def install_environment(prefix, version, additional_dependencies): -# """Installs a repository in the given repository. Note that the current -# working directory will already be inside the repository. -# -# Args: -# prefix - `Prefix` bound to the repository. -# version - A version specified in the hook configuration or 'default'. -# """ -# -# def run_hook(hook, file_args): -# """Runs a hook and returns the returncode and output of running that -# hook. -# -# Args: -# hook - `Hook` -# file_args - The files to be run -# -# Returns: -# (returncode, stdout, stderr) -# """ - -languages = { - 'docker': docker, - 'docker_image': docker_image, - 'fail': fail, - 'golang': golang, - 'node': node, - 'pcre': pcre, - 'pygrep': pygrep, - 'python': python, - 'python_venv': python_venv, - 'ruby': ruby, - 'rust': rust, - 'script': script, - 'swift': swift, - 'system': system, -} -all_languages = sorted(languages) diff --git a/pre_commit/languages/conda.py b/pre_commit/languages/conda.py new file mode 100644 index 00000000..d397ebeb --- /dev/null +++ b/pre_commit/languages/conda.py @@ -0,0 +1,77 @@ +from __future__ import annotations + +import contextlib +import os +import sys +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import SubstitutionT +from pre_commit.envcontext import UNSET +from pre_commit.envcontext import Var +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output_b + +ENVIRONMENT_DIR = 'conda' +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def get_env_patch(env: str) -> PatchesT: + # On non-windows systems executable live in $CONDA_PREFIX/bin, on Windows + # they can be in $CONDA_PREFIX/bin, $CONDA_PREFIX/Library/bin, + # $CONDA_PREFIX/Scripts and $CONDA_PREFIX. Whereas the latter only + # seems to be used for python.exe. + path: SubstitutionT = (os.path.join(env, 'bin'), os.pathsep, Var('PATH')) + if sys.platform == 'win32': # pragma: win32 cover + path = (env, os.pathsep, *path) + path = (os.path.join(env, 'Scripts'), os.pathsep, *path) + path = (os.path.join(env, 'Library', 'bin'), os.pathsep, *path) + + return ( + ('PYTHONHOME', UNSET), + ('VIRTUAL_ENV', UNSET), + ('CONDA_PREFIX', env), + ('PATH', path), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +def _conda_exe() -> str: + if os.environ.get('PRE_COMMIT_USE_MICROMAMBA'): + return 'micromamba' + elif os.environ.get('PRE_COMMIT_USE_MAMBA'): + return 'mamba' + else: + return 'conda' + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('conda', version) + + conda_exe = _conda_exe() + + env_dir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + cmd_output_b( + conda_exe, 'env', 'create', '-p', env_dir, '--file', + 'environment.yml', cwd=prefix.prefix_dir, + ) + if additional_dependencies: + cmd_output_b( + conda_exe, 'install', '-p', env_dir, *additional_dependencies, + cwd=prefix.prefix_dir, + ) diff --git a/pre_commit/languages/coursier.py b/pre_commit/languages/coursier.py new file mode 100644 index 00000000..08f9a958 --- /dev/null +++ b/pre_commit/languages/coursier.py @@ -0,0 +1,76 @@ +from __future__ import annotations + +import contextlib +import os.path +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import Var +from pre_commit.errors import FatalError +from pre_commit.parse_shebang import find_executable +from pre_commit.prefix import Prefix + +ENVIRONMENT_DIR = 'coursier' + +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('coursier', version) + + # Support both possible executable names (either "cs" or "coursier") + cs = find_executable('cs') or find_executable('coursier') + if cs is None: + raise AssertionError( + 'pre-commit requires system-installed "cs" or "coursier" ' + 'executables in the application search path', + ) + + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + + def _install(*opts: str) -> None: + assert cs is not None + lang_base.setup_cmd(prefix, (cs, 'fetch', *opts)) + lang_base.setup_cmd(prefix, (cs, 'install', '--dir', envdir, *opts)) + + with in_env(prefix, version): + channel = prefix.path('.pre-commit-channel') + if os.path.isdir(channel): + for app_descriptor in os.listdir(channel): + _, app_file = os.path.split(app_descriptor) + app, _ = os.path.splitext(app_file) + _install( + '--default-channels=false', + '--channel', channel, + app, + ) + elif not additional_dependencies: + raise FatalError( + 'expected .pre-commit-channel dir or additional_dependencies', + ) + + if additional_dependencies: + _install(*additional_dependencies) + + +def get_env_patch(target_dir: str) -> PatchesT: + return ( + ('PATH', (target_dir, os.pathsep, Var('PATH'))), + ('COURSIER_CACHE', os.path.join(target_dir, '.cs-cache')), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield diff --git a/pre_commit/languages/dart.py b/pre_commit/languages/dart.py new file mode 100644 index 00000000..52a229ee --- /dev/null +++ b/pre_commit/languages/dart.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import contextlib +import os.path +import shutil +import tempfile +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import Var +from pre_commit.prefix import Prefix +from pre_commit.util import win_exe +from pre_commit.yaml import yaml_load + +ENVIRONMENT_DIR = 'dartenv' + +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def get_env_patch(venv: str) -> PatchesT: + return ( + ('PATH', (os.path.join(venv, 'bin'), os.pathsep, Var('PATH'))), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('dart', version) + + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + bin_dir = os.path.join(envdir, 'bin') + + def _install_dir(prefix_p: Prefix, pub_cache: str) -> None: + dart_env = {**os.environ, 'PUB_CACHE': pub_cache} + + with open(prefix_p.path('pubspec.yaml')) as f: + pubspec_contents = yaml_load(f) + + lang_base.setup_cmd(prefix_p, ('dart', 'pub', 'get'), env=dart_env) + + for executable in pubspec_contents['executables']: + lang_base.setup_cmd( + prefix_p, + ( + 'dart', 'compile', 'exe', + '--output', os.path.join(bin_dir, win_exe(executable)), + prefix_p.path('bin', f'{executable}.dart'), + ), + env=dart_env, + ) + + os.makedirs(bin_dir) + + with tempfile.TemporaryDirectory() as tmp: + _install_dir(prefix, tmp) + + for dep_s in additional_dependencies: + with tempfile.TemporaryDirectory() as dep_tmp: + dep, _, version = dep_s.partition(':') + if version: + dep_cmd: tuple[str, ...] = (dep, '--version', version) + else: + dep_cmd = (dep,) + + lang_base.setup_cmd( + prefix, + ('dart', 'pub', 'cache', 'add', *dep_cmd), + env={**os.environ, 'PUB_CACHE': dep_tmp}, + ) + + # try and find the 'pubspec.yaml' that just got added + for root, _, filenames in os.walk(dep_tmp): + if 'pubspec.yaml' in filenames: + with tempfile.TemporaryDirectory() as copied: + pkg = os.path.join(copied, 'pkg') + shutil.copytree(root, pkg) + _install_dir(Prefix(pkg), dep_tmp) + break + else: + raise AssertionError( + f'could not find pubspec.yaml for {dep_s}', + ) diff --git a/pre_commit/languages/docker.py b/pre_commit/languages/docker.py index 4517050b..7f45ac86 100644 --- a/pre_commit/languages/docker.py +++ b/pre_commit/languages/docker.py @@ -1,49 +1,86 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations +import contextlib +import functools import hashlib +import json import os +import re +from collections.abc import Sequence -import pre_commit.constants as C -from pre_commit import five -from pre_commit.languages import helpers +from pre_commit import lang_base +from pre_commit.prefix import Prefix from pre_commit.util import CalledProcessError -from pre_commit.util import clean_path_on_failure -from pre_commit.util import cmd_output - +from pre_commit.util import cmd_output_b ENVIRONMENT_DIR = 'docker' PRE_COMMIT_LABEL = 'PRE_COMMIT' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +in_env = lang_base.no_env # no special environment for docker + +_HOSTNAME_MOUNT_RE = re.compile( + rb""" + /containers + (?:/overlay-containers)? + /([a-z0-9]{64}) + (?:/userdata)? + /hostname + """, + re.VERBOSE, +) -def md5(s): # pragma: windows no cover - return hashlib.md5(five.to_bytes(s)).hexdigest() +def _get_container_id() -> str | None: + with contextlib.suppress(FileNotFoundError): + with open('/proc/1/mountinfo', 'rb') as f: + for line in f: + m = _HOSTNAME_MOUNT_RE.search(line) + if m: + return m[1].decode() + + return None -def docker_tag(prefix): # pragma: windows no cover - md5sum = md5(os.path.basename(prefix.prefix_dir)).lower() - return 'pre-commit-{}'.format(md5sum) +def _get_docker_path(path: str) -> str: + container_id = _get_container_id() + if container_id is None: + return path - -def docker_is_running(): # pragma: windows no cover try: - return cmd_output('docker', 'ps')[0] == 0 + _, out, _ = cmd_output_b('docker', 'inspect', container_id) except CalledProcessError: - return False + # self-container was not visible from here (perhaps docker-in-docker) + return path + + container, = json.loads(out) + for mount in container['Mounts']: + src_path = mount['Source'] + to_path = mount['Destination'] + if os.path.commonpath((path, to_path)) == to_path: + # So there is something in common, + # and we can proceed remapping it + return path.replace(to_path, src_path) + # we're in Docker, but the path is not mounted, cannot really do anything, + # so fall back to original path + return path -def assert_docker_available(): # pragma: windows no cover - assert docker_is_running(), ( - 'Docker is either not running or not configured in this environment' - ) +def md5(s: str) -> str: # pragma: win32 no cover + return hashlib.md5(s.encode()).hexdigest() -def build_docker_image(prefix, **kwargs): # pragma: windows no cover - pull = kwargs.pop('pull') - assert not kwargs, kwargs - cmd = ( +def docker_tag(prefix: Prefix) -> str: # pragma: win32 no cover + md5sum = md5(os.path.basename(prefix.prefix_dir)).lower() + return f'pre-commit-{md5sum}' + + +def build_docker_image( + prefix: Prefix, + *, + pull: bool, +) -> None: # pragma: win32 no cover + cmd: tuple[str, ...] = ( 'docker', 'build', '--tag', docker_tag(prefix), '--label', PRE_COMMIT_LABEL, @@ -52,56 +89,93 @@ def build_docker_image(prefix, **kwargs): # pragma: windows no cover cmd += ('--pull',) # This must come last for old versions of docker. See #477 cmd += ('.',) - helpers.run_setup_cmd(prefix, cmd) + lang_base.setup_cmd(prefix, cmd) def install_environment( - prefix, version, additional_dependencies, -): # pragma: windows no cover - helpers.assert_version_default('docker', version) - helpers.assert_no_additional_deps('docker', additional_dependencies) - assert_docker_available() + prefix: Prefix, version: str, additional_dependencies: Sequence[str], +) -> None: # pragma: win32 no cover + lang_base.assert_version_default('docker', version) + lang_base.assert_no_additional_deps('docker', additional_dependencies) - directory = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) + directory = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) # Docker doesn't really have relevant disk environment, but pre-commit - # still needs to cleanup it's state files on failure - with clean_path_on_failure(directory): - build_docker_image(prefix, pull=True) - os.mkdir(directory) + # still needs to cleanup its state files on failure + build_docker_image(prefix, pull=True) + os.mkdir(directory) -def get_docker_user(): # pragma: windows no cover +@functools.lru_cache(maxsize=1) +def _is_rootless() -> bool: # pragma: win32 no cover + retcode, out, _ = cmd_output_b( + 'docker', 'system', 'info', '--format', '{{ json . }}', + ) + if retcode != 0: + return False + + info = json.loads(out) try: - return '{}:{}'.format(os.getuid(), os.getgid()) + return ( + # docker: + # https://docs.docker.com/reference/api/engine/version/v1.48/#tag/System/operation/SystemInfo + 'name=rootless' in (info.get('SecurityOptions') or ()) or + # podman: + # https://docs.podman.io/en/latest/_static/api.html?version=v5.4#tag/system/operation/SystemInfoLibpod + info['host']['security']['rootless'] + ) + except KeyError: + return False + + +def get_docker_user() -> tuple[str, ...]: # pragma: win32 no cover + if _is_rootless(): + return () + + try: + return ('-u', f'{os.getuid()}:{os.getgid()}') except AttributeError: - return '1000:1000' + return () -def docker_cmd(): # pragma: windows no cover +def get_docker_tty(*, color: bool) -> tuple[str, ...]: # pragma: win32 no cover # noqa: E501 + return (('--tty',) if color else ()) + + +def docker_cmd(*, color: bool) -> tuple[str, ...]: # pragma: win32 no cover return ( 'docker', 'run', '--rm', - '-u', get_docker_user(), + *get_docker_tty(color=color), + *get_docker_user(), # https://docs.docker.com/engine/reference/commandline/run/#mount-volumes-from-container-volumes-from # The `Z` option tells Docker to label the content with a private # unshared label. Only the current container can use a private volume. - '-v', '{}:/src:rw,Z'.format(os.getcwd()), + '-v', f'{_get_docker_path(os.getcwd())}:/src:rw,Z', '--workdir', '/src', ) -def run_hook(hook, file_args): # pragma: windows no cover - assert_docker_available() +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: # pragma: win32 no cover # Rebuild the docker image in case it has gone missing, as many people do # automated cleanup of docker images. - build_docker_image(hook.prefix, pull=False) + build_docker_image(prefix, pull=False) - hook_cmd = helpers.to_cmd(hook) - entry_exe, cmd_rest = hook_cmd[0], hook_cmd[1:] + entry_exe, *cmd_rest = lang_base.hook_cmd(entry, args) - entry_tag = ('--entrypoint', entry_exe, docker_tag(hook.prefix)) - cmd = docker_cmd() + entry_tag + cmd_rest - return helpers.run_xargs(hook, cmd, file_args) + entry_tag = ('--entrypoint', entry_exe, docker_tag(prefix)) + return lang_base.run_xargs( + (*docker_cmd(color=color), *entry_tag, *cmd_rest), + file_args, + require_serial=require_serial, + color=color, + ) diff --git a/pre_commit/languages/docker_image.py b/pre_commit/languages/docker_image.py index ab2a8565..60caa101 100644 --- a/pre_commit/languages/docker_image.py +++ b/pre_commit/languages/docker_image.py @@ -1,18 +1,32 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -from pre_commit.languages import helpers -from pre_commit.languages.docker import assert_docker_available +from collections.abc import Sequence + +from pre_commit import lang_base from pre_commit.languages.docker import docker_cmd - +from pre_commit.prefix import Prefix ENVIRONMENT_DIR = None -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy -install_environment = helpers.no_install +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +install_environment = lang_base.no_install +in_env = lang_base.no_env -def run_hook(hook, file_args): # pragma: windows no cover - assert_docker_available() - cmd = docker_cmd() + helpers.to_cmd(hook) - return helpers.run_xargs(hook, cmd, file_args) +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: # pragma: win32 no cover + cmd = docker_cmd(color=color) + lang_base.hook_cmd(entry, args) + return lang_base.run_xargs( + cmd, + file_args, + require_serial=require_serial, + color=color, + ) diff --git a/pre_commit/languages/dotnet.py b/pre_commit/languages/dotnet.py new file mode 100644 index 00000000..ffc65d1e --- /dev/null +++ b/pre_commit/languages/dotnet.py @@ -0,0 +1,111 @@ +from __future__ import annotations + +import contextlib +import os.path +import re +import tempfile +import xml.etree.ElementTree +import zipfile +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import Var +from pre_commit.prefix import Prefix + +ENVIRONMENT_DIR = 'dotnetenv' +BIN_DIR = 'bin' + +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def get_env_patch(venv: str) -> PatchesT: + return ( + ('PATH', (os.path.join(venv, BIN_DIR), os.pathsep, Var('PATH'))), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +@contextlib.contextmanager +def _nuget_config_no_sources() -> Generator[str]: + with tempfile.TemporaryDirectory() as tmpdir: + nuget_config = os.path.join(tmpdir, 'nuget.config') + with open(nuget_config, 'w') as f: + f.write( + '' + '' + ' ' + ' ' + ' ' + '', + ) + yield nuget_config + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('dotnet', version) + lang_base.assert_no_additional_deps('dotnet', additional_dependencies) + + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + build_dir = prefix.path('pre-commit-build') + + # Build & pack nupkg file + lang_base.setup_cmd( + prefix, + ( + 'dotnet', 'pack', + '--configuration', 'Release', + '--property', f'PackageOutputPath={build_dir}', + ), + ) + + nupkg_dir = prefix.path(build_dir) + nupkgs = [x for x in os.listdir(nupkg_dir) if x.endswith('.nupkg')] + + if not nupkgs: + raise AssertionError('could not find any build outputs to install') + + for nupkg in nupkgs: + with zipfile.ZipFile(os.path.join(nupkg_dir, nupkg)) as f: + nuspec, = (x for x in f.namelist() if x.endswith('.nuspec')) + with f.open(nuspec) as spec: + tree = xml.etree.ElementTree.parse(spec) + + namespace = re.match(r'{.*}', tree.getroot().tag) + if not namespace: + raise AssertionError('could not parse namespace from nuspec') + + tool_id_element = tree.find(f'.//{namespace[0]}id') + if tool_id_element is None: + raise AssertionError('expected to find an "id" element') + + tool_id = tool_id_element.text + if not tool_id: + raise AssertionError('"id" element missing tool name') + + # Install to bin dir + with _nuget_config_no_sources() as nuget_config: + lang_base.setup_cmd( + prefix, + ( + 'dotnet', 'tool', 'install', + '--configfile', nuget_config, + '--tool-path', os.path.join(envdir, BIN_DIR), + '--add-source', build_dir, + tool_id, + ), + ) diff --git a/pre_commit/languages/fail.py b/pre_commit/languages/fail.py index f2ce09e1..6ac4d767 100644 --- a/pre_commit/languages/fail.py +++ b/pre_commit/languages/fail.py @@ -1,15 +1,27 @@ -from __future__ import unicode_literals +from __future__ import annotations -from pre_commit.languages import helpers +from collections.abc import Sequence +from pre_commit import lang_base +from pre_commit.prefix import Prefix ENVIRONMENT_DIR = None -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy -install_environment = helpers.no_install +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +install_environment = lang_base.no_install +in_env = lang_base.no_env -def run_hook(hook, file_args): - out = hook.entry.encode('UTF-8') + b'\n\n' - out += b'\n'.join(f.encode('UTF-8') for f in file_args) + b'\n' - return 1, out, b'' +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + out = f'{entry}\n\n'.encode() + out += b'\n'.join(f.encode() for f in file_args) + b'\n' + return 1, out diff --git a/pre_commit/languages/golang.py b/pre_commit/languages/golang.py index f6124dd5..bedbd114 100644 --- a/pre_commit/languages/golang.py +++ b/pre_commit/languages/golang.py @@ -1,85 +1,161 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib +import functools +import json import os.path +import platform +import shutil import sys +import tarfile +import tempfile +import urllib.error +import urllib.request +import zipfile +from collections.abc import Generator +from collections.abc import Sequence +from typing import ContextManager +from typing import IO +from typing import Protocol import pre_commit.constants as C -from pre_commit import git +from pre_commit import lang_base from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT from pre_commit.envcontext import Var -from pre_commit.languages import helpers -from pre_commit.util import clean_path_on_failure +from pre_commit.git import no_git_env +from pre_commit.prefix import Prefix from pre_commit.util import cmd_output from pre_commit.util import rmtree - ENVIRONMENT_DIR = 'golangenv' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + +_ARCH_ALIASES = { + 'x86_64': 'amd64', + 'i386': '386', + 'aarch64': 'arm64', + 'armv8': 'arm64', + 'armv7l': 'armv6l', +} +_ARCH = platform.machine().lower() +_ARCH = _ARCH_ALIASES.get(_ARCH, _ARCH) -def get_env_patch(venv): +class ExtractAll(Protocol): + def extractall(self, path: str) -> None: ... + + +if sys.platform == 'win32': # pragma: win32 cover + _EXT = 'zip' + + def _open_archive(bio: IO[bytes]) -> ContextManager[ExtractAll]: + return zipfile.ZipFile(bio) +else: # pragma: win32 no cover + _EXT = 'tar.gz' + + def _open_archive(bio: IO[bytes]) -> ContextManager[ExtractAll]: + return tarfile.open(fileobj=bio) + + +@functools.lru_cache(maxsize=1) +def get_default_version() -> str: + if lang_base.exe_exists('go'): + return 'system' + else: + return C.DEFAULT + + +def get_env_patch(venv: str, version: str) -> PatchesT: + if version == 'system': + return ( + ('PATH', (os.path.join(venv, 'bin'), os.pathsep, Var('PATH'))), + ) + return ( - ('PATH', (os.path.join(venv, 'bin'), os.pathsep, Var('PATH'))), + ('GOROOT', os.path.join(venv, '.go')), + ('GOTOOLCHAIN', 'local'), + ( + 'PATH', ( + os.path.join(venv, 'bin'), os.pathsep, + os.path.join(venv, '.go', 'bin'), os.pathsep, Var('PATH'), + ), + ), ) +@functools.lru_cache +def _infer_go_version(version: str) -> str: + if version != C.DEFAULT: + return version + resp = urllib.request.urlopen('https://go.dev/dl/?mode=json') + return json.load(resp)[0]['version'].removeprefix('go') + + +def _get_url(version: str) -> str: + os_name = platform.system().lower() + version = _infer_go_version(version) + return f'https://dl.google.com/go/go{version}.{os_name}-{_ARCH}.{_EXT}' + + +def _install_go(version: str, dest: str) -> None: + try: + resp = urllib.request.urlopen(_get_url(version)) + except urllib.error.HTTPError as e: # pragma: no cover + if e.code == 404: + raise ValueError( + f'Could not find a version matching your system requirements ' + f'(os={platform.system().lower()}; arch={_ARCH})', + ) from e + else: + raise + else: + with tempfile.TemporaryFile() as f: + shutil.copyfileobj(resp, f) + f.seek(0) + + with _open_archive(f) as archive: + archive.extractall(dest) + shutil.move(os.path.join(dest, 'go'), os.path.join(dest, '.go')) + + @contextlib.contextmanager -def in_env(prefix): - envdir = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) - with envcontext(get_env_patch(envdir)): +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir, version)): yield -def guess_go_dir(remote_url): - if remote_url.endswith('.git'): - remote_url = remote_url[:-1 * len('.git')] - looks_like_url = ( - not remote_url.startswith('file://') and - ('//' in remote_url or '@' in remote_url) - ) - remote_url = remote_url.replace(':', '/') - if looks_like_url: - _, _, remote_url = remote_url.rpartition('//') - _, _, remote_url = remote_url.rpartition('@') - return remote_url +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + env_dir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + + if version != 'system': + _install_go(version, env_dir) + + if sys.platform == 'cygwin': # pragma: no cover + gopath = cmd_output('cygpath', '-w', env_dir)[1].strip() else: - return 'unknown_src_dir' + gopath = env_dir + env = no_git_env(dict(os.environ, GOPATH=gopath)) + env.pop('GOBIN', None) + if version != 'system': + env['GOTOOLCHAIN'] = 'local' + env['GOROOT'] = os.path.join(env_dir, '.go') + env['PATH'] = os.pathsep.join(( + os.path.join(env_dir, '.go', 'bin'), os.environ['PATH'], + )) -def install_environment(prefix, version, additional_dependencies): - helpers.assert_version_default('golang', version) - directory = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) + lang_base.setup_cmd(prefix, ('go', 'install', './...'), env=env) + for dependency in additional_dependencies: + lang_base.setup_cmd(prefix, ('go', 'install', dependency), env=env) - with clean_path_on_failure(directory): - remote = git.get_remote_url(prefix.prefix_dir) - repo_src_dir = os.path.join(directory, 'src', guess_go_dir(remote)) - - # Clone into the goenv we'll create - helpers.run_setup_cmd(prefix, ('git', 'clone', '.', repo_src_dir)) - - if sys.platform == 'cygwin': # pragma: no cover - _, gopath, _ = cmd_output('cygpath', '-w', directory) - gopath = gopath.strip() - else: - gopath = directory - env = dict(os.environ, GOPATH=gopath) - env.pop('GOBIN', None) - cmd_output('go', 'get', './...', cwd=repo_src_dir, env=env) - for dependency in additional_dependencies: - cmd_output('go', 'get', dependency, cwd=repo_src_dir, env=env) - # Same some disk space, we don't need these after installation - rmtree(prefix.path(directory, 'src')) - pkgdir = prefix.path(directory, 'pkg') - if os.path.exists(pkgdir): # pragma: no cover (go<1.10) - rmtree(pkgdir) - - -def run_hook(hook, file_args): - with in_env(hook.prefix): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) + # save some disk space -- we don't need this after installation + pkgdir = os.path.join(env_dir, 'pkg') + if os.path.exists(pkgdir): # pragma: no branch (always true on windows?) + rmtree(pkgdir) diff --git a/pre_commit/languages/haskell.py b/pre_commit/languages/haskell.py new file mode 100644 index 00000000..28bca08c --- /dev/null +++ b/pre_commit/languages/haskell.py @@ -0,0 +1,56 @@ +from __future__ import annotations + +import contextlib +import os.path +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import Var +from pre_commit.errors import FatalError +from pre_commit.prefix import Prefix + +ENVIRONMENT_DIR = 'hs_env' +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def get_env_patch(target_dir: str) -> PatchesT: + bin_path = os.path.join(target_dir, 'bin') + return (('PATH', (bin_path, os.pathsep, Var('PATH'))),) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('haskell', version) + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + + pkgs = [*prefix.star('.cabal'), *additional_dependencies] + if not pkgs: + raise FatalError('Expected .cabal files or additional_dependencies') + + bindir = os.path.join(envdir, 'bin') + os.makedirs(bindir, exist_ok=True) + lang_base.setup_cmd(prefix, ('cabal', 'update')) + lang_base.setup_cmd( + prefix, + ( + 'cabal', 'install', + '--install-method', 'copy', + '--installdir', bindir, + *pkgs, + ), + ) diff --git a/pre_commit/languages/helpers.py b/pre_commit/languages/helpers.py deleted file mode 100644 index 0915f410..00000000 --- a/pre_commit/languages/helpers.py +++ /dev/null @@ -1,90 +0,0 @@ -from __future__ import unicode_literals - -import multiprocessing -import os -import random -import shlex - -import six - -import pre_commit.constants as C -from pre_commit.util import cmd_output -from pre_commit.xargs import xargs - -FIXED_RANDOM_SEED = 1542676186 - - -def run_setup_cmd(prefix, cmd): - cmd_output(*cmd, cwd=prefix.prefix_dir, encoding=None) - - -def environment_dir(ENVIRONMENT_DIR, language_version): - if ENVIRONMENT_DIR is None: - return None - else: - return '{}-{}'.format(ENVIRONMENT_DIR, language_version) - - -def to_cmd(hook): - return tuple(shlex.split(hook.entry)) + tuple(hook.args) - - -def assert_version_default(binary, version): - if version != C.DEFAULT: - raise AssertionError( - 'For now, pre-commit requires system-installed {}'.format(binary), - ) - - -def assert_no_additional_deps(lang, additional_deps): - if additional_deps: - raise AssertionError( - 'For now, pre-commit does not support ' - 'additional_dependencies for {}'.format(lang), - ) - - -def basic_get_default_version(): - return C.DEFAULT - - -def basic_healthy(prefix, language_version): - return True - - -def no_install(prefix, version, additional_dependencies): - raise AssertionError('This type is not installable') - - -def target_concurrency(hook): - if hook.require_serial or 'PRE_COMMIT_NO_CONCURRENCY' in os.environ: - return 1 - else: - # Travis appears to have a bunch of CPUs, but we can't use them all. - if 'TRAVIS' in os.environ: - return 2 - else: - try: - return multiprocessing.cpu_count() - except NotImplementedError: - return 1 - - -def _shuffled(seq): - """Deterministically shuffle identically under both py2 + py3.""" - fixed_random = random.Random() - if six.PY2: # pragma: no cover (py2) - fixed_random.seed(FIXED_RANDOM_SEED) - else: # pragma: no cover (py3) - fixed_random.seed(FIXED_RANDOM_SEED, version=1) - - seq = list(seq) - random.shuffle(seq, random=fixed_random.random) - return seq - - -def run_xargs(hook, cmd, file_args): - # Shuffle the files so that they more evenly fill out the xargs partitions, - # but do it deterministically in case a hook cares about ordering. - file_args = _shuffled(file_args) - return xargs(cmd, file_args, target_concurrency=target_concurrency(hook)) diff --git a/pre_commit/languages/julia.py b/pre_commit/languages/julia.py new file mode 100644 index 00000000..7559b5ba --- /dev/null +++ b/pre_commit/languages/julia.py @@ -0,0 +1,133 @@ +from __future__ import annotations + +import contextlib +import os +import shutil +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import UNSET +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output_b + +ENVIRONMENT_DIR = 'juliaenv' +health_check = lang_base.basic_health_check +get_default_version = lang_base.basic_get_default_version + + +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + # `entry` is a (hook-repo relative) file followed by (optional) args, e.g. + # `bin/id.jl` or `bin/hook.jl --arg1 --arg2` so we + # 1) shell parse it and join with args with hook_cmd + # 2) prepend the hooks prefix path to the first argument (the file), unless + # it is a local script + # 3) prepend `julia` as the interpreter + + cmd = lang_base.hook_cmd(entry, args) + script = cmd[0] if is_local else prefix.path(cmd[0]) + cmd = ('julia', '--startup-file=no', script, *cmd[1:]) + return lang_base.run_xargs( + cmd, + file_args, + require_serial=require_serial, + color=color, + ) + + +def get_env_patch(target_dir: str, version: str) -> PatchesT: + return ( + ('JULIA_LOAD_PATH', target_dir), + # May be set, remove it to not interfer with LOAD_PATH + ('JULIA_PROJECT', UNSET), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir, version)): + yield + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with in_env(prefix, version): + # TODO: Support language_version with juliaup similar to rust via + # rustup + # if version != 'system': + # ... + + # Copy Project.toml to hook env if it exist + os.makedirs(envdir, exist_ok=True) + project_names = ('JuliaProject.toml', 'Project.toml') + project_found = False + for project_name in project_names: + project_file = prefix.path(project_name) + if not os.path.isfile(project_file): + continue + shutil.copy(project_file, envdir) + project_found = True + break + + # If no project file was found we create an empty one so that the + # package manager doesn't error + if not project_found: + open(os.path.join(envdir, 'Project.toml'), 'a').close() + + # Copy Manifest.toml to hook env if it exists + manifest_names = ('JuliaManifest.toml', 'Manifest.toml') + for manifest_name in manifest_names: + manifest_file = prefix.path(manifest_name) + if not os.path.isfile(manifest_file): + continue + shutil.copy(manifest_file, envdir) + break + + # Julia code to instantiate the hook environment + julia_code = """ + @assert length(ARGS) > 0 + hook_env = ARGS[1] + deps = join(ARGS[2:end], " ") + + # We prepend @stdlib here so that we can load the package manager even + # though `get_env_patch` limits `JULIA_LOAD_PATH` to just the hook env. + pushfirst!(LOAD_PATH, "@stdlib") + using Pkg + popfirst!(LOAD_PATH) + + # Instantiate the environment shipped with the hook repo. If we have + # additional dependencies we disable precompilation in this step to + # avoid double work. + precompile = isempty(deps) ? "1" : "0" + withenv("JULIA_PKG_PRECOMPILE_AUTO" => precompile) do + Pkg.instantiate() + end + + # Add additional dependencies (with precompilation) + if !isempty(deps) + withenv("JULIA_PKG_PRECOMPILE_AUTO" => "1") do + Pkg.REPLMode.pkgstr("add " * deps) + end + end + """ + cmd_output_b( + 'julia', '--startup-file=no', '-e', julia_code, '--', envdir, + *additional_dependencies, + cwd=prefix.prefix_dir, + ) diff --git a/pre_commit/languages/lua.py b/pre_commit/languages/lua.py new file mode 100644 index 00000000..15ac1a2e --- /dev/null +++ b/pre_commit/languages/lua.py @@ -0,0 +1,75 @@ +from __future__ import annotations + +import contextlib +import os +import sys +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import Var +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output + +ENVIRONMENT_DIR = 'lua_env' +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def _get_lua_version() -> str: # pragma: win32 no cover + """Get the Lua version used in file paths.""" + _, stdout, _ = cmd_output('luarocks', 'config', '--lua-ver') + return stdout.strip() + + +def get_env_patch(d: str) -> PatchesT: # pragma: win32 no cover + version = _get_lua_version() + so_ext = 'dll' if sys.platform == 'win32' else 'so' + return ( + ('PATH', (os.path.join(d, 'bin'), os.pathsep, Var('PATH'))), + ( + 'LUA_PATH', ( + os.path.join(d, 'share', 'lua', version, '?.lua;'), + os.path.join(d, 'share', 'lua', version, '?', 'init.lua;;'), + ), + ), + ( + 'LUA_CPATH', + (os.path.join(d, 'lib', 'lua', version, f'?.{so_ext};;'),), + ), + ) + + +@contextlib.contextmanager # pragma: win32 no cover +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: # pragma: win32 no cover + lang_base.assert_version_default('lua', version) + + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with in_env(prefix, version): + # luarocks doesn't bootstrap a tree prior to installing + # so ensure the directory exists. + os.makedirs(envdir, exist_ok=True) + + # Older luarocks (e.g., 2.4.2) expect the rockspec as an arg + for rockspec in prefix.star('.rockspec'): + make_cmd = ('luarocks', '--tree', envdir, 'make', rockspec) + lang_base.setup_cmd(prefix, make_cmd) + + # luarocks can't install multiple packages at once + # so install them individually. + for dependency in additional_dependencies: + cmd = ('luarocks', '--tree', envdir, 'install', dependency) + lang_base.setup_cmd(prefix, cmd) diff --git a/pre_commit/languages/node.py b/pre_commit/languages/node.py index cd3b7b54..af7dc6f8 100644 --- a/pre_commit/languages/node.py +++ b/pre_commit/languages/node.py @@ -1,79 +1,110 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib +import functools import os import sys +from collections.abc import Generator +from collections.abc import Sequence import pre_commit.constants as C +from pre_commit import lang_base from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import UNSET from pre_commit.envcontext import Var -from pre_commit.languages import helpers from pre_commit.languages.python import bin_dir -from pre_commit.util import clean_path_on_failure +from pre_commit.prefix import Prefix from pre_commit.util import cmd_output - +from pre_commit.util import cmd_output_b +from pre_commit.util import rmtree ENVIRONMENT_DIR = 'node_env' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy +run_hook = lang_base.basic_run_hook -def _envdir(prefix, version): - directory = helpers.environment_dir(ENVIRONMENT_DIR, version) - return prefix.path(directory) +@functools.lru_cache(maxsize=1) +def get_default_version() -> str: + # nodeenv does not yet support `-n system` on windows + if sys.platform == 'win32': + return C.DEFAULT + # if node is already installed, we can save a bunch of setup time by + # using the installed version + elif all(lang_base.exe_exists(exe) for exe in ('node', 'npm')): + return 'system' + else: + return C.DEFAULT -def get_env_patch(venv): # pragma: windows no cover +def get_env_patch(venv: str) -> PatchesT: if sys.platform == 'cygwin': # pragma: no cover _, win_venv, _ = cmd_output('cygpath', '-w', venv) - install_prefix = r'{}\bin'.format(win_venv.strip()) + install_prefix = fr'{win_venv.strip()}\bin' + lib_dir = 'lib' elif sys.platform == 'win32': # pragma: no cover install_prefix = bin_dir(venv) - else: # pragma: windows no cover + lib_dir = 'Scripts' + else: # pragma: win32 no cover install_prefix = venv + lib_dir = 'lib' return ( ('NODE_VIRTUAL_ENV', venv), ('NPM_CONFIG_PREFIX', install_prefix), ('npm_config_prefix', install_prefix), - ('NODE_PATH', os.path.join(venv, 'lib', 'node_modules')), + ('NPM_CONFIG_USERCONFIG', UNSET), + ('npm_config_userconfig', UNSET), + ('NODE_PATH', os.path.join(venv, lib_dir, 'node_modules')), ('PATH', (bin_dir(venv), os.pathsep, Var('PATH'))), ) @contextlib.contextmanager -def in_env(prefix, language_version): # pragma: windows no cover - with envcontext(get_env_patch(_envdir(prefix, language_version))): +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): yield +def health_check(prefix: Prefix, version: str) -> str | None: + with in_env(prefix, version): + retcode, _, _ = cmd_output_b('node', '--version', check=False) + if retcode != 0: # pragma: win32 no cover + return f'`node --version` returned {retcode}' + else: + return None + + def install_environment( - prefix, version, additional_dependencies, -): # pragma: windows no cover - additional_dependencies = tuple(additional_dependencies) + prefix: Prefix, version: str, additional_dependencies: Sequence[str], +) -> None: assert prefix.exists('package.json') - envdir = _envdir(prefix, version) + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) # https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx?f=255&MSPPError=-2147217396#maxpath if sys.platform == 'win32': # pragma: no cover - envdir = '\\\\?\\' + os.path.normpath(envdir) - with clean_path_on_failure(envdir): - cmd = [ - sys.executable, '-mnodeenv', '--prebuilt', '--clean-src', envdir, - ] - if version != C.DEFAULT: - cmd.extend(['-n', version]) - cmd_output(*cmd) + envdir = fr'\\?\{os.path.normpath(envdir)}' + cmd = [sys.executable, '-mnodeenv', '--prebuilt', '--clean-src', envdir] + if version != C.DEFAULT: + cmd.extend(['-n', version]) + cmd_output_b(*cmd) - with in_env(prefix, version): - # https://npm.community/t/npm-install-g-git-vs-git-clone-cd-npm-install-g/5449 - # install as if we installed from git - helpers.run_setup_cmd(prefix, ('npm', 'install')) - helpers.run_setup_cmd( - prefix, - ('npm', 'install', '-g', '.') + additional_dependencies, - ) + with in_env(prefix, version): + # https://npm.community/t/npm-install-g-git-vs-git-clone-cd-npm-install-g/5449 + # install as if we installed from git + local_install_cmd = ( + 'npm', 'install', '--include=dev', '--include=prod', + '--ignore-prepublish', '--no-progress', '--no-save', + ) + lang_base.setup_cmd(prefix, local_install_cmd) -def run_hook(hook, file_args): # pragma: windows no cover - with in_env(hook.prefix, hook.language_version): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) + _, pkg, _ = cmd_output('npm', 'pack', cwd=prefix.prefix_dir) + pkg = prefix.path(pkg.strip()) + + install = ('npm', 'install', '-g', pkg, *additional_dependencies) + lang_base.setup_cmd(prefix, install) + + # clean these up after installation + if prefix.exists('node_modules'): # pragma: win32 no cover + rmtree(prefix.path('node_modules')) + os.remove(pkg) diff --git a/pre_commit/languages/pcre.py b/pre_commit/languages/pcre.py deleted file mode 100644 index 143adb23..00000000 --- a/pre_commit/languages/pcre.py +++ /dev/null @@ -1,22 +0,0 @@ -from __future__ import unicode_literals - -import sys - -from pre_commit.languages import helpers -from pre_commit.xargs import xargs - - -ENVIRONMENT_DIR = None -GREP = 'ggrep' if sys.platform == 'darwin' else 'grep' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy -install_environment = helpers.no_install - - -def run_hook(hook, file_args): - # For PCRE the entry is the regular expression to match - cmd = (GREP, '-H', '-n', '-P') + tuple(hook.args) + (hook.entry,) - - # Grep usually returns 0 for matches, and nonzero for non-matches so we - # negate it here. - return xargs(cmd, file_args, negate=True) diff --git a/pre_commit/languages/perl.py b/pre_commit/languages/perl.py new file mode 100644 index 00000000..a07d442a --- /dev/null +++ b/pre_commit/languages/perl.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +import contextlib +import os +import shlex +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import Var +from pre_commit.prefix import Prefix + +ENVIRONMENT_DIR = 'perl_env' +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook + + +def get_env_patch(venv: str) -> PatchesT: + return ( + ('PATH', (os.path.join(venv, 'bin'), os.pathsep, Var('PATH'))), + ('PERL5LIB', os.path.join(venv, 'lib', 'perl5')), + ('PERL_MB_OPT', f'--install_base {shlex.quote(venv)}'), + ( + 'PERL_MM_OPT', ( + f'INSTALL_BASE={shlex.quote(venv)} ' + f'INSTALLSITEMAN1DIR=none INSTALLSITEMAN3DIR=none' + ), + ), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +def install_environment( + prefix: Prefix, version: str, additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('perl', version) + + with in_env(prefix, version): + lang_base.setup_cmd( + prefix, ('cpan', '-T', '.', *additional_dependencies), + ) diff --git a/pre_commit/languages/pygrep.py b/pre_commit/languages/pygrep.py index e0188a97..72a9345f 100644 --- a/pre_commit/languages/pygrep.py +++ b/pre_commit/languages/pygrep.py @@ -1,33 +1,36 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import argparse import re import sys +from collections.abc import Sequence +from re import Pattern +from typing import NamedTuple +from pre_commit import lang_base from pre_commit import output -from pre_commit.languages import helpers +from pre_commit.prefix import Prefix from pre_commit.xargs import xargs - ENVIRONMENT_DIR = None -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy -install_environment = helpers.no_install +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +install_environment = lang_base.no_install +in_env = lang_base.no_env -def _process_filename_by_line(pattern, filename): +def _process_filename_by_line(pattern: Pattern[bytes], filename: str) -> int: retv = 0 with open(filename, 'rb') as f: for line_no, line in enumerate(f, start=1): if pattern.search(line): retv = 1 - output.write('{}:{}:'.format(filename, line_no)) - output.write_line(line.rstrip(b'\r\n')) + output.write(f'{filename}:{line_no}:') + output.write_line_b(line.rstrip(b'\r\n')) return retv -def _process_filename_at_once(pattern, filename): +def _process_filename_at_once(pattern: Pattern[bytes], filename: str) -> int: retv = 0 with open(filename, 'rb') as f: contents = f.read() @@ -35,21 +38,70 @@ def _process_filename_at_once(pattern, filename): if match: retv = 1 line_no = contents[:match.start()].count(b'\n') - output.write('{}:{}:'.format(filename, line_no + 1)) + output.write(f'{filename}:{line_no + 1}:') - matched_lines = match.group().split(b'\n') + matched_lines = match[0].split(b'\n') matched_lines[0] = contents.split(b'\n')[line_no] - output.write_line(b'\n'.join(matched_lines)) + output.write_line_b(b'\n'.join(matched_lines)) return retv -def run_hook(hook, file_args): - exe = (sys.executable, '-m', __name__) + tuple(hook.args) + (hook.entry,) - return xargs(exe, file_args) +def _process_filename_by_line_negated( + pattern: Pattern[bytes], + filename: str, +) -> int: + with open(filename, 'rb') as f: + for line in f: + if pattern.search(line): + return 0 + else: + output.write_line(filename) + return 1 -def main(argv=None): +def _process_filename_at_once_negated( + pattern: Pattern[bytes], + filename: str, +) -> int: + with open(filename, 'rb') as f: + contents = f.read() + match = pattern.search(contents) + if match: + return 0 + else: + output.write_line(filename) + return 1 + + +class Choice(NamedTuple): + multiline: bool + negate: bool + + +FNS = { + Choice(multiline=True, negate=True): _process_filename_at_once_negated, + Choice(multiline=True, negate=False): _process_filename_at_once, + Choice(multiline=False, negate=True): _process_filename_by_line_negated, + Choice(multiline=False, negate=False): _process_filename_by_line, +} + + +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + cmd = (sys.executable, '-m', __name__, *args, entry) + return xargs(cmd, file_args, color=color) + + +def main(argv: Sequence[str] | None = None) -> int: parser = argparse.ArgumentParser( description=( 'grep-like finder using python regexes. Unlike grep, this tool ' @@ -59,6 +111,7 @@ def main(argv=None): ) parser.add_argument('-i', '--ignore-case', action='store_true') parser.add_argument('--multiline', action='store_true') + parser.add_argument('--negate', action='store_true') parser.add_argument('pattern', help='python regex pattern.') parser.add_argument('filenames', nargs='*') args = parser.parse_args(argv) @@ -70,13 +123,11 @@ def main(argv=None): pattern = re.compile(args.pattern.encode(), flags) retv = 0 + process_fn = FNS[Choice(multiline=args.multiline, negate=args.negate)] for filename in args.filenames: - if args.multiline: - retv |= _process_filename_at_once(pattern, filename) - else: - retv |= _process_filename_by_line(pattern, filename) + retv |= process_fn(pattern, filename) return retv if __name__ == '__main__': - exit(main()) + raise SystemExit(main()) diff --git a/pre_commit/languages/python.py b/pre_commit/languages/python.py index 5d48fb89..88ececce 100644 --- a/pre_commit/languages/python.py +++ b/pre_commit/languages/python.py @@ -1,56 +1,95 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib +import functools import os import sys +from collections.abc import Generator +from collections.abc import Sequence import pre_commit.constants as C +from pre_commit import lang_base from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT from pre_commit.envcontext import UNSET from pre_commit.envcontext import Var -from pre_commit.languages import helpers from pre_commit.parse_shebang import find_executable +from pre_commit.prefix import Prefix from pre_commit.util import CalledProcessError -from pre_commit.util import clean_path_on_failure from pre_commit.util import cmd_output - +from pre_commit.util import cmd_output_b +from pre_commit.util import win_exe ENVIRONMENT_DIR = 'py_env' +run_hook = lang_base.basic_run_hook -def bin_dir(venv): +@functools.cache +def _version_info(exe: str) -> str: + prog = 'import sys;print(".".join(str(p) for p in sys.version_info))' + try: + return cmd_output(exe, '-S', '-c', prog)[1].strip() + except CalledProcessError: + return f'<>' + + +def _read_pyvenv_cfg(filename: str) -> dict[str, str]: + ret = {} + with open(filename, encoding='UTF-8') as f: + for line in f: + try: + k, v = line.split('=') + except ValueError: # blank line / comment / etc. + continue + else: + ret[k.strip()] = v.strip() + return ret + + +def bin_dir(venv: str) -> str: """On windows there's a different directory for the virtualenv""" - bin_part = 'Scripts' if os.name == 'nt' else 'bin' + bin_part = 'Scripts' if sys.platform == 'win32' else 'bin' return os.path.join(venv, bin_part) -def get_env_patch(venv): +def get_env_patch(venv: str) -> PatchesT: return ( + ('PIP_DISABLE_PIP_VERSION_CHECK', '1'), ('PYTHONHOME', UNSET), ('VIRTUAL_ENV', venv), ('PATH', (bin_dir(venv), os.pathsep, Var('PATH'))), ) -def _find_by_py_launcher(version): # pragma: no cover (windows only) +def _find_by_py_launcher( + version: str, +) -> str | None: # pragma: no cover (windows only) if version.startswith('python'): + num = version.removeprefix('python') + cmd = ('py', f'-{num}', '-c', 'import sys; print(sys.executable)') + env = dict(os.environ, PYTHONIOENCODING='UTF-8') try: - return cmd_output( - 'py', '-{}'.format(version[len('python'):]), - '-c', 'import sys; print(sys.executable)', - )[1].strip() + return cmd_output(*cmd, env=env)[1].strip() except CalledProcessError: pass + return None -def _get_default_version(): # pragma: no cover (platform dependent) - def _norm(path): +def _impl_exe_name() -> str: + if sys.implementation.name == 'cpython': # pragma: cpython cover + return 'python' + else: # pragma: cpython no cover + return sys.implementation.name # pypy mostly + + +def _find_by_sys_executable() -> str | None: + def _norm(path: str) -> str | None: _, exe = os.path.split(path.lower()) exe, _, _ = exe.partition('.exe') - if find_executable(exe) and exe not in {'python', 'pythonw'}: + if exe not in {'python', 'pythonw'} and find_executable(exe): return exe + return None - # First attempt from `sys.executable` (or the realpath) # On linux, I see these common sys.executables: # # system `python`: /usr/bin/python -> python2.7 @@ -59,56 +98,60 @@ def _get_default_version(): # pragma: no cover (platform dependent) # virtualenv v -ppython2: v/bin/python -> python2 # virtualenv v -ppython2.7: v/bin/python -> python2.7 # virtualenv v -ppypy: v/bin/python -> v/bin/pypy - for path in {sys.executable, os.path.realpath(sys.executable)}: + for path in (sys.executable, os.path.realpath(sys.executable)): exe = _norm(path) if exe: return exe + return None - # Next try the `pythonX.X` executable - exe = 'python{}.{}'.format(*sys.version_info) - if find_executable(exe): - return exe - if _find_by_py_launcher(exe): - return exe +@functools.lru_cache(maxsize=1) +def get_default_version() -> str: # pragma: no cover (platform dependent) + v_major = f'{sys.version_info[0]}' + v_minor = f'{sys.version_info[0]}.{sys.version_info[1]}' - # Give a best-effort try for windows - if os.path.exists(r'C:\{}\python.exe'.format(exe.replace('.', ''))): - return exe + # attempt the likely implementation exe + for potential in (v_minor, v_major): + exe = f'{_impl_exe_name()}{potential}' + if find_executable(exe): + return exe + + # next try `sys.executable` (or the realpath) + maybe_exe = _find_by_sys_executable() + if maybe_exe: + return maybe_exe + + # maybe on windows we can find it via py launcher? + if sys.platform == 'win32': # pragma: win32 cover + exe = f'python{v_minor}' + if _find_by_py_launcher(exe): + return exe # We tried! return C.DEFAULT -def get_default_version(): - # TODO: when dropping python2, use `functools.lru_cache(maxsize=1)` - try: - return get_default_version.cached_version - except AttributeError: - get_default_version.cached_version = _get_default_version() - return get_default_version() - - -def _sys_executable_matches(version): +def _sys_executable_matches(version: str) -> bool: if version == 'python': return True elif not version.startswith('python'): return False try: - info = tuple(int(p) for p in version[len('python'):].split('.')) + info = tuple(int(p) for p in version.removeprefix('python').split('.')) except ValueError: return False return sys.version_info[:len(info)] == info -def norm_version(version): - # first see if our current executable is appropriate - if _sys_executable_matches(version): - return sys.executable +def norm_version(version: str) -> str | None: + if version == C.DEFAULT: # use virtualenv's default + return None + elif _sys_executable_matches(version): # virtualenv defaults to our exe + return None - if os.name == 'nt': # pragma: no cover (windows) + if sys.platform == 'win32': # pragma: no cover (windows) version_exec = _find_by_py_launcher(version) if version_exec: return version_exec @@ -118,60 +161,68 @@ def norm_version(version): if version_exec and version_exec != version: return version_exec - # If it is in the form pythonx.x search in the default - # place on windows - if version.startswith('python'): - return r'C:\{}\python.exe'.format(version.replace('.', '')) - # Otherwise assume it is a path return os.path.expanduser(version) -def py_interface(_dir, _make_venv): - @contextlib.contextmanager - def in_env(prefix, language_version): - envdir = prefix.path(helpers.environment_dir(_dir, language_version)) - with envcontext(get_env_patch(envdir)): - yield - - def healthy(prefix, language_version): - with in_env(prefix, language_version): - retcode, _, _ = cmd_output( - 'python', '-c', - 'import ctypes, datetime, io, os, ssl, weakref', - retcode=None, - encoding=None, - ) - return retcode == 0 - - def run_hook(hook, file_args): - with in_env(hook.prefix, hook.language_version): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) - - def install_environment(prefix, version, additional_dependencies): - additional_dependencies = tuple(additional_dependencies) - directory = helpers.environment_dir(_dir, version) - - env_dir = prefix.path(directory) - with clean_path_on_failure(env_dir): - if version != C.DEFAULT: - python = norm_version(version) - else: - python = os.path.realpath(sys.executable) - _make_venv(env_dir, python) - with in_env(prefix, version): - helpers.run_setup_cmd( - prefix, ('pip', 'install', '.') + additional_dependencies, - ) - - return in_env, healthy, run_hook, install_environment +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield -def make_venv(envdir, python): - env = dict(os.environ, VIRTUALENV_NO_DOWNLOAD='1') - cmd = (sys.executable, '-mvirtualenv', envdir, '-p', python) - cmd_output(*cmd, env=env, cwd='/') +def health_check(prefix: Prefix, version: str) -> str | None: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + pyvenv_cfg = os.path.join(envdir, 'pyvenv.cfg') + + # created with "old" virtualenv + if not os.path.exists(pyvenv_cfg): + return 'pyvenv.cfg does not exist (old virtualenv?)' + + exe_name = win_exe('python') + py_exe = prefix.path(bin_dir(envdir), exe_name) + cfg = _read_pyvenv_cfg(pyvenv_cfg) + + if 'version_info' not in cfg: + return "created virtualenv's pyvenv.cfg is missing `version_info`" + + # always use uncached lookup here in case we replaced an unhealthy env + virtualenv_version = _version_info.__wrapped__(py_exe) + if virtualenv_version != cfg['version_info']: + return ( + f'virtualenv python version did not match created version:\n' + f'- actual version: {virtualenv_version}\n' + f'- expected version: {cfg["version_info"]}\n' + ) + + # made with an older version of virtualenv? skip `base-executable` check + if 'base-executable' not in cfg: + return None + + base_exe_version = _version_info(cfg['base-executable']) + if base_exe_version != cfg['version_info']: + return ( + f'base executable python version does not match created version:\n' + f'- base-executable version: {base_exe_version}\n' + f'- expected version: {cfg["version_info"]}\n' + ) + else: + return None -_interface = py_interface(ENVIRONMENT_DIR, make_venv) -in_env, healthy, run_hook, install_environment = _interface +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + venv_cmd = [sys.executable, '-mvirtualenv', envdir] + python = norm_version(version) + if python is not None: + venv_cmd.extend(('-p', python)) + install_cmd = ('python', '-mpip', 'install', '.', *additional_dependencies) + + cmd_output_b(*venv_cmd, cwd='/') + with in_env(prefix, version): + lang_base.setup_cmd(prefix, install_cmd) diff --git a/pre_commit/languages/python_venv.py b/pre_commit/languages/python_venv.py deleted file mode 100644 index b7658f5d..00000000 --- a/pre_commit/languages/python_venv.py +++ /dev/null @@ -1,55 +0,0 @@ -from __future__ import unicode_literals - -import os.path -import sys - -from pre_commit.languages import python -from pre_commit.util import CalledProcessError -from pre_commit.util import cmd_output - - -ENVIRONMENT_DIR = 'py_venv' - - -def get_default_version(): # pragma: no cover (version specific) - if sys.version_info < (3,): - return 'python3' - else: - return python.get_default_version() - - -def orig_py_exe(exe): # pragma: no cover (platform specific) - """A -mvenv virtualenv made from a -mvirtualenv virtualenv installs - packages to the incorrect location. Attempt to find the _original_ exe - and invoke `-mvenv` from there. - - See: - - https://github.com/pre-commit/pre-commit/issues/755 - - https://github.com/pypa/virtualenv/issues/1095 - - https://bugs.python.org/issue30811 - """ - try: - prefix_script = 'import sys; print(sys.real_prefix)' - _, prefix, _ = cmd_output(exe, '-c', prefix_script) - prefix = prefix.strip() - except CalledProcessError: - # not created from -mvirtualenv - return exe - - if os.name == 'nt': - expected = os.path.join(prefix, 'python.exe') - else: - expected = os.path.join(prefix, 'bin', os.path.basename(exe)) - - if os.path.exists(expected): - return expected - else: - return exe - - -def make_venv(envdir, python): - cmd_output(orig_py_exe(python), '-mvenv', envdir, cwd='/') - - -_interface = python.py_interface(ENVIRONMENT_DIR, make_venv) -in_env, healthy, run_hook, install_environment = _interface diff --git a/pre_commit/languages/r.py b/pre_commit/languages/r.py new file mode 100644 index 00000000..f70d2fdc --- /dev/null +++ b/pre_commit/languages/r.py @@ -0,0 +1,278 @@ +from __future__ import annotations + +import contextlib +import os +import shlex +import shutil +import tempfile +import textwrap +from collections.abc import Generator +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import UNSET +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output +from pre_commit.util import win_exe + +ENVIRONMENT_DIR = 'renv' +get_default_version = lang_base.basic_get_default_version + +_RENV_ACTIVATED_OPTS = ( + '--no-save', '--no-restore', '--no-site-file', '--no-environ', +) + + +def _execute_r( + code: str, *, + prefix: Prefix, version: str, args: Sequence[str] = (), cwd: str, + cli_opts: Sequence[str], +) -> str: + with in_env(prefix, version), _r_code_in_tempfile(code) as f: + _, out, _ = cmd_output( + _rscript_exec(), *cli_opts, f, *args, cwd=cwd, + ) + return out.rstrip('\n') + + +def _execute_r_in_renv( + code: str, *, + prefix: Prefix, version: str, args: Sequence[str] = (), cwd: str, +) -> str: + return _execute_r( + code=code, prefix=prefix, version=version, args=args, cwd=cwd, + cli_opts=_RENV_ACTIVATED_OPTS, + ) + + +def _execute_vanilla_r( + code: str, *, + prefix: Prefix, version: str, args: Sequence[str] = (), cwd: str, +) -> str: + return _execute_r( + code=code, prefix=prefix, version=version, args=args, cwd=cwd, + cli_opts=('--vanilla',), + ) + + +def _read_installed_version(envdir: str, prefix: Prefix, version: str) -> str: + return _execute_r_in_renv( + 'cat(renv::settings$r.version())', + prefix=prefix, version=version, + cwd=envdir, + ) + + +def _read_executable_version(envdir: str, prefix: Prefix, version: str) -> str: + return _execute_r_in_renv( + 'cat(as.character(getRversion()))', + prefix=prefix, version=version, + cwd=envdir, + ) + + +def _write_current_r_version( + envdir: str, prefix: Prefix, version: str, +) -> None: + _execute_r_in_renv( + 'renv::settings$r.version(as.character(getRversion()))', + prefix=prefix, version=version, + cwd=envdir, + ) + + +def health_check(prefix: Prefix, version: str) -> str | None: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + + r_version_installation = _read_installed_version( + envdir=envdir, prefix=prefix, version=version, + ) + r_version_current_executable = _read_executable_version( + envdir=envdir, prefix=prefix, version=version, + ) + if r_version_installation in {'NULL', ''}: + return ( + f'Hooks were installed with an unknown R version. R version for ' + f'hook repo now set to {r_version_current_executable}' + ) + elif r_version_installation != r_version_current_executable: + return ( + f'Hooks were installed for R version {r_version_installation}, ' + f'but current R executable has version ' + f'{r_version_current_executable}' + ) + + return None + + +@contextlib.contextmanager +def _r_code_in_tempfile(code: str) -> Generator[str]: + """ + To avoid quoting and escaping issues, avoid `Rscript [options] -e {expr}` + but use `Rscript [options] path/to/file_with_expr.R` + """ + with tempfile.TemporaryDirectory() as tmpdir: + fname = os.path.join(tmpdir, 'script.R') + with open(fname, 'w') as f: + f.write(_inline_r_setup(textwrap.dedent(code))) + yield fname + + +def get_env_patch(venv: str) -> PatchesT: + return ( + ('R_PROFILE_USER', os.path.join(venv, 'activate.R')), + ('RENV_PROJECT', UNSET), + ) + + +@contextlib.contextmanager +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir)): + yield + + +def _prefix_if_file_entry( + entry: list[str], + prefix: Prefix, + *, + is_local: bool, +) -> Sequence[str]: + if entry[1] == '-e' or is_local: + return entry[1:] + else: + return (prefix.path(entry[1]),) + + +def _rscript_exec() -> str: + r_home = os.environ.get('R_HOME') + if r_home is None: + return 'Rscript' + else: + return os.path.join(r_home, 'bin', win_exe('Rscript')) + + +def _entry_validate(entry: list[str]) -> None: + """ + Allowed entries: + # Rscript -e expr + # Rscript path/to/file + """ + if entry[0] != 'Rscript': + raise ValueError('entry must start with `Rscript`.') + + if entry[1] == '-e': + if len(entry) > 3: + raise ValueError('You can supply at most one expression.') + elif len(entry) > 2: + raise ValueError( + 'The only valid syntax is `Rscript -e {expr}`' + 'or `Rscript path/to/hook/script`', + ) + + +def _cmd_from_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + *, + is_local: bool, +) -> tuple[str, ...]: + cmd = shlex.split(entry) + _entry_validate(cmd) + + cmd_part = _prefix_if_file_entry(cmd, prefix, is_local=is_local) + return (cmd[0], *_RENV_ACTIVATED_OPTS, *cmd_part, *args) + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + lang_base.assert_version_default('r', version) + + env_dir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + os.makedirs(env_dir, exist_ok=True) + shutil.copy(prefix.path('renv.lock'), env_dir) + shutil.copytree(prefix.path('renv'), os.path.join(env_dir, 'renv')) + + r_code_inst_environment = f"""\ + prefix_dir <- {prefix.prefix_dir!r} + options( + repos = c(CRAN = "https://cran.rstudio.com"), + renv.consent = TRUE + ) + source("renv/activate.R") + renv::restore() + activate_statement <- paste0( + 'suppressWarnings({{', + 'old <- setwd("', getwd(), '"); ', + 'source("renv/activate.R"); ', + 'setwd(old); ', + 'renv::load("', getwd(), '");}})' + ) + writeLines(activate_statement, 'activate.R') + is_package <- tryCatch( + {{ + path_desc <- file.path(prefix_dir, 'DESCRIPTION') + suppressWarnings(desc <- read.dcf(path_desc)) + "Package" %in% colnames(desc) + }}, + error = function(...) FALSE + ) + if (is_package) {{ + renv::install(prefix_dir) + }} + """ + _execute_vanilla_r( + r_code_inst_environment, + prefix=prefix, version=version, cwd=env_dir, + ) + + _write_current_r_version(envdir=env_dir, prefix=prefix, version=version) + if additional_dependencies: + r_code_inst_add = 'renv::install(commandArgs(trailingOnly = TRUE))' + _execute_r_in_renv( + code=r_code_inst_add, prefix=prefix, version=version, + args=additional_dependencies, + cwd=env_dir, + ) + + +def _inline_r_setup(code: str) -> str: + """ + Some behaviour of R cannot be configured via env variables, but can + only be configured via R options once R has started. These are set here. + """ + with_option = [ + textwrap.dedent("""\ + options( + install.packages.compile.from.source = "never", + pkgType = "binary" + ) + """), + code, + ] + return '\n'.join(with_option) + + +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + cmd = _cmd_from_hook(prefix, entry, args, is_local=is_local) + return lang_base.run_xargs( + cmd, + file_args, + require_serial=require_serial, + color=color, + ) diff --git a/pre_commit/languages/ruby.py b/pre_commit/languages/ruby.py index c721b3ce..f32fea3f 100644 --- a/pre_commit/languages/ruby.py +++ b/pre_commit/languages/ruby.py @@ -1,129 +1,145 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib -import io +import functools +import importlib.resources import os.path import shutil import tarfile +from collections.abc import Generator +from collections.abc import Sequence +from typing import IO import pre_commit.constants as C +from pre_commit import lang_base from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT +from pre_commit.envcontext import UNSET from pre_commit.envcontext import Var -from pre_commit.languages import helpers +from pre_commit.prefix import Prefix from pre_commit.util import CalledProcessError -from pre_commit.util import clean_path_on_failure -from pre_commit.util import resource_bytesio - ENVIRONMENT_DIR = 'rbenv' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook -def get_env_patch(venv, language_version): # pragma: windows no cover - patches = ( +def _resource_bytesio(filename: str) -> IO[bytes]: + files = importlib.resources.files('pre_commit.resources') + return files.joinpath(filename).open('rb') + + +@functools.lru_cache(maxsize=1) +def get_default_version() -> str: + if all(lang_base.exe_exists(exe) for exe in ('ruby', 'gem')): + return 'system' + else: + return C.DEFAULT + + +def get_env_patch( + venv: str, + language_version: str, +) -> PatchesT: + patches: PatchesT = ( ('GEM_HOME', os.path.join(venv, 'gems')), - ('RBENV_ROOT', venv), + ('GEM_PATH', UNSET), ('BUNDLE_IGNORE_CONFIG', '1'), - ( - 'PATH', ( - os.path.join(venv, 'gems', 'bin'), os.pathsep, - os.path.join(venv, 'shims'), os.pathsep, - os.path.join(venv, 'bin'), os.pathsep, Var('PATH'), - ), - ), ) - if language_version != C.DEFAULT: + if language_version == 'system': + patches += ( + ( + 'PATH', ( + os.path.join(venv, 'gems', 'bin'), os.pathsep, + Var('PATH'), + ), + ), + ) + else: # pragma: win32 no cover + patches += ( + ('RBENV_ROOT', venv), + ( + 'PATH', ( + os.path.join(venv, 'gems', 'bin'), os.pathsep, + os.path.join(venv, 'shims'), os.pathsep, + os.path.join(venv, 'bin'), os.pathsep, Var('PATH'), + ), + ), + ) + if language_version not in {'system', 'default'}: # pragma: win32 no cover patches += (('RBENV_VERSION', language_version),) + return patches @contextlib.contextmanager -def in_env(prefix, language_version): # pragma: windows no cover - envdir = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, language_version), - ) - with envcontext(get_env_patch(envdir, language_version)): +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir, version)): yield -def _extract_resource(filename, dest): - with resource_bytesio(filename) as bio: +def _extract_resource(filename: str, dest: str) -> None: + with _resource_bytesio(filename) as bio: with tarfile.open(fileobj=bio) as tf: tf.extractall(dest) -def _install_rbenv(prefix, version=C.DEFAULT): # pragma: windows no cover - directory = helpers.environment_dir(ENVIRONMENT_DIR, version) +def _install_rbenv( + prefix: Prefix, + version: str, +) -> None: # pragma: win32 no cover + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) _extract_resource('rbenv.tar.gz', prefix.path('.')) - shutil.move(prefix.path('rbenv'), prefix.path(directory)) + shutil.move(prefix.path('rbenv'), envdir) # Only install ruby-build if the version is specified if version != C.DEFAULT: - plugins_dir = prefix.path(directory, 'plugins') + plugins_dir = os.path.join(envdir, 'plugins') _extract_resource('ruby-download.tar.gz', plugins_dir) _extract_resource('ruby-build.tar.gz', plugins_dir) - activate_path = prefix.path(directory, 'bin', 'activate') - with io.open(activate_path, 'w') as activate_file: - # This is similar to how you would install rbenv to your home directory - # However we do a couple things to make the executables exposed and - # configure it to work in our directory. - # We also modify the PS1 variable for manual debugging sake. - activate_file.write( - '#!/usr/bin/env bash\n' - "export RBENV_ROOT='{directory}'\n" - 'export PATH="$RBENV_ROOT/bin:$PATH"\n' - 'eval "$(rbenv init -)"\n' - 'export PS1="(rbenv)$PS1"\n' - # This lets us install gems in an isolated and repeatable - # directory - "export GEM_HOME='{directory}/gems'\n" - 'export PATH="$GEM_HOME/bin:$PATH"\n' - '\n'.format(directory=prefix.path(directory)), - ) - # If we aren't using the system ruby, add a version here - if version != C.DEFAULT: - activate_file.write('export RBENV_VERSION="{}"\n'.format(version)) - - -def _install_ruby(prefix, version): # pragma: windows no cover +def _install_ruby( + prefix: Prefix, + version: str, +) -> None: # pragma: win32 no cover try: - helpers.run_setup_cmd(prefix, ('rbenv', 'download', version)) + lang_base.setup_cmd(prefix, ('rbenv', 'download', version)) except CalledProcessError: # pragma: no cover (usually find with download) # Failed to download from mirror for some reason, build it instead - helpers.run_setup_cmd(prefix, ('rbenv', 'install', version)) + lang_base.setup_cmd(prefix, ('rbenv', 'install', version)) def install_environment( - prefix, version, additional_dependencies, -): # pragma: windows no cover - additional_dependencies = tuple(additional_dependencies) - directory = helpers.environment_dir(ENVIRONMENT_DIR, version) - with clean_path_on_failure(prefix.path(directory)): - # TODO: this currently will fail if there's no version specified and - # there's no system ruby installed. Is this ok? - _install_rbenv(prefix, version=version) + prefix: Prefix, version: str, additional_dependencies: Sequence[str], +) -> None: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + + if version != 'system': # pragma: win32 no cover + _install_rbenv(prefix, version) with in_env(prefix, version): - # Need to call this before installing so rbenv's directories are - # set up - helpers.run_setup_cmd(prefix, ('rbenv', 'init', '-')) + # Need to call this before installing so rbenv's directories + # are set up + lang_base.setup_cmd(prefix, ('rbenv', 'init', '-')) if version != C.DEFAULT: _install_ruby(prefix, version) # Need to call this after installing to set up the shims - helpers.run_setup_cmd(prefix, ('rbenv', 'rehash')) - helpers.run_setup_cmd( - prefix, ('gem', 'build') + prefix.star('.gemspec'), - ) - helpers.run_setup_cmd( - prefix, - ('gem', 'install', '--no-document') + - prefix.star('.gem') + additional_dependencies, - ) + lang_base.setup_cmd(prefix, ('rbenv', 'rehash')) - -def run_hook(hook, file_args): # pragma: windows no cover - with in_env(hook.prefix, hook.language_version): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) + with in_env(prefix, version): + lang_base.setup_cmd( + prefix, ('gem', 'build', *prefix.star('.gemspec')), + ) + lang_base.setup_cmd( + prefix, + ( + 'gem', 'install', + '--no-document', '--no-format-executable', + '--no-user-install', + '--install-dir', os.path.join(envdir, 'gems'), + '--bindir', os.path.join(envdir, 'gems', 'bin'), + *prefix.star('.gem'), *additional_dependencies, + ), + ) diff --git a/pre_commit/languages/rust.py b/pre_commit/languages/rust.py index 4b25a9d1..fd77a9d2 100644 --- a/pre_commit/languages/rust.py +++ b/pre_commit/languages/rust.py @@ -1,58 +1,121 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib +import functools import os.path - -import toml +import shutil +import sys +import tempfile +import urllib.request +from collections.abc import Generator +from collections.abc import Sequence import pre_commit.constants as C +from pre_commit import lang_base +from pre_commit import parse_shebang from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT from pre_commit.envcontext import Var -from pre_commit.languages import helpers -from pre_commit.util import clean_path_on_failure -from pre_commit.util import cmd_output - +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output_b +from pre_commit.util import make_executable +from pre_commit.util import win_exe ENVIRONMENT_DIR = 'rustenv' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook -def get_env_patch(target_dir): +@functools.lru_cache(maxsize=1) +def get_default_version() -> str: + # If rust is already installed, we can save a bunch of setup time by + # using the installed version. + # + # Just detecting the executable does not suffice, because if rustup is + # installed but no toolchain is available, then `cargo` exists but + # cannot be used without installing a toolchain first. + if cmd_output_b('cargo', '--version', check=False, cwd='/')[0] == 0: + return 'system' + else: + return C.DEFAULT + + +def _rust_toolchain(language_version: str) -> str: + """Transform the language version into a rust toolchain version.""" + if language_version == C.DEFAULT: + return 'stable' + else: + return language_version + + +def get_env_patch(target_dir: str, version: str) -> PatchesT: return ( - ( - 'PATH', - (os.path.join(target_dir, 'bin'), os.pathsep, Var('PATH')), + ('PATH', (os.path.join(target_dir, 'bin'), os.pathsep, Var('PATH'))), + # Only set RUSTUP_TOOLCHAIN if we don't want use the system's default + # toolchain + *( + (('RUSTUP_TOOLCHAIN', _rust_toolchain(version)),) + if version != 'system' else () ), ) @contextlib.contextmanager -def in_env(prefix): - target_dir = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) - with envcontext(get_env_patch(target_dir)): +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) + with envcontext(get_env_patch(envdir, version)): yield -def _add_dependencies(cargo_toml_path, additional_dependencies): - with open(cargo_toml_path, 'r+') as f: - cargo_toml = toml.load(f) - cargo_toml.setdefault('dependencies', {}) - for dep in additional_dependencies: - name, _, spec = dep.partition(':') - cargo_toml['dependencies'][name] = spec or '*' - f.seek(0) - toml.dump(cargo_toml, f) - f.truncate() +def _add_dependencies( + prefix: Prefix, + additional_dependencies: set[str], +) -> None: + crates = [] + for dep in additional_dependencies: + name, _, spec = dep.partition(':') + crate = f'{name}@{spec or "*"}' + crates.append(crate) + + lang_base.setup_cmd(prefix, ('cargo', 'add', *crates)) -def install_environment(prefix, version, additional_dependencies): - helpers.assert_version_default('rust', version) - directory = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) +def install_rust_with_toolchain(toolchain: str, envdir: str) -> None: + with tempfile.TemporaryDirectory() as rustup_dir: + with envcontext((('CARGO_HOME', envdir), ('RUSTUP_HOME', rustup_dir))): + # acquire `rustup` if not present + if parse_shebang.find_executable('rustup') is None: + # We did not detect rustup and need to download it first. + if sys.platform == 'win32': # pragma: win32 cover + url = 'https://win.rustup.rs/x86_64' + else: # pragma: win32 no cover + url = 'https://sh.rustup.rs' + + resp = urllib.request.urlopen(url) + + rustup_init = os.path.join(rustup_dir, win_exe('rustup-init')) + with open(rustup_init, 'wb') as f: + shutil.copyfileobj(resp, f) + make_executable(rustup_init) + + # install rustup into `$CARGO_HOME/bin` + cmd_output_b( + rustup_init, '-y', '--quiet', '--no-modify-path', + '--default-toolchain', 'none', + ) + + cmd_output_b( + 'rustup', 'toolchain', 'install', '--no-self-update', + toolchain, + ) + + +def install_environment( + prefix: Prefix, + version: str, + additional_dependencies: Sequence[str], +) -> None: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) # There are two cases where we might want to specify more dependencies: # as dependencies for the library being built, and as binary packages @@ -69,26 +132,29 @@ def install_environment(prefix, version, additional_dependencies): } lib_deps = set(additional_dependencies) - cli_deps - if len(lib_deps) > 0: - _add_dependencies(prefix.path('Cargo.toml'), lib_deps) + packages_to_install: set[tuple[str, ...]] = {('--path', '.')} + for cli_dep in cli_deps: + cli_dep = cli_dep.removeprefix('cli:') + package, _, crate_version = cli_dep.partition(':') + if crate_version != '': + packages_to_install.add((package, '--version', crate_version)) + else: + packages_to_install.add((package,)) - with clean_path_on_failure(directory): - packages_to_install = {('--path', '.')} - for cli_dep in cli_deps: - cli_dep = cli_dep[len('cli:'):] - package, _, version = cli_dep.partition(':') - if version != '': - packages_to_install.add((package, '--version', version)) - else: - packages_to_install.add((package,)) + with contextlib.ExitStack() as ctx: + ctx.enter_context(in_env(prefix, version)) - for package in packages_to_install: - cmd_output( - 'cargo', 'install', '--bins', '--root', directory, *package, - cwd=prefix.prefix_dir + if version != 'system': + install_rust_with_toolchain(_rust_toolchain(version), envdir) + + tmpdir = ctx.enter_context(tempfile.TemporaryDirectory()) + ctx.enter_context(envcontext((('RUSTUP_HOME', tmpdir),))) + + if len(lib_deps) > 0: + _add_dependencies(prefix, lib_deps) + + for args in packages_to_install: + cmd_output_b( + 'cargo', 'install', '--bins', '--root', envdir, *args, + cwd=prefix.prefix_dir, ) - - -def run_hook(hook, file_args): - with in_env(hook.prefix): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) diff --git a/pre_commit/languages/script.py b/pre_commit/languages/script.py deleted file mode 100644 index 56d9d27e..00000000 --- a/pre_commit/languages/script.py +++ /dev/null @@ -1,15 +0,0 @@ -from __future__ import unicode_literals - -from pre_commit.languages import helpers - - -ENVIRONMENT_DIR = None -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy -install_environment = helpers.no_install - - -def run_hook(hook, file_args): - cmd = helpers.to_cmd(hook) - cmd = (hook.prefix.path(cmd[0]),) + cmd[1:] - return helpers.run_xargs(hook, cmd, file_args) diff --git a/pre_commit/languages/swift.py b/pre_commit/languages/swift.py index 3f5a92f1..08a9c39a 100644 --- a/pre_commit/languages/swift.py +++ b/pre_commit/languages/swift.py @@ -1,56 +1,50 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib import os +from collections.abc import Generator +from collections.abc import Sequence -import pre_commit.constants as C +from pre_commit import lang_base from pre_commit.envcontext import envcontext +from pre_commit.envcontext import PatchesT from pre_commit.envcontext import Var -from pre_commit.languages import helpers -from pre_commit.util import clean_path_on_failure -from pre_commit.util import cmd_output +from pre_commit.prefix import Prefix +from pre_commit.util import cmd_output_b -ENVIRONMENT_DIR = 'swift_env' -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy BUILD_DIR = '.build' BUILD_CONFIG = 'release' +ENVIRONMENT_DIR = 'swift_env' +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +run_hook = lang_base.basic_run_hook -def get_env_patch(venv): # pragma: windows no cover + +def get_env_patch(venv: str) -> PatchesT: # pragma: win32 no cover bin_path = os.path.join(venv, BUILD_DIR, BUILD_CONFIG) return (('PATH', (bin_path, os.pathsep, Var('PATH'))),) -@contextlib.contextmanager -def in_env(prefix): # pragma: windows no cover - envdir = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) +@contextlib.contextmanager # pragma: win32 no cover +def in_env(prefix: Prefix, version: str) -> Generator[None]: + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) with envcontext(get_env_patch(envdir)): yield def install_environment( - prefix, version, additional_dependencies, -): # pragma: windows no cover - helpers.assert_version_default('swift', version) - helpers.assert_no_additional_deps('swift', additional_dependencies) - directory = prefix.path( - helpers.environment_dir(ENVIRONMENT_DIR, C.DEFAULT), - ) + prefix: Prefix, version: str, additional_dependencies: Sequence[str], +) -> None: # pragma: win32 no cover + lang_base.assert_version_default('swift', version) + lang_base.assert_no_additional_deps('swift', additional_dependencies) + envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version) # Build the swift package - with clean_path_on_failure(directory): - os.mkdir(directory) - cmd_output( - 'swift', 'build', - '-C', prefix.prefix_dir, - '-c', BUILD_CONFIG, - '--build-path', os.path.join(directory, BUILD_DIR), - ) - - -def run_hook(hook, file_args): # pragma: windows no cover - with in_env(hook.prefix): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) + os.mkdir(envdir) + cmd_output_b( + 'swift', 'build', + '--package-path', prefix.prefix_dir, + '-c', BUILD_CONFIG, + '--build-path', os.path.join(envdir, BUILD_DIR), + ) diff --git a/pre_commit/languages/system.py b/pre_commit/languages/system.py deleted file mode 100644 index 5a22670e..00000000 --- a/pre_commit/languages/system.py +++ /dev/null @@ -1,13 +0,0 @@ -from __future__ import unicode_literals - -from pre_commit.languages import helpers - - -ENVIRONMENT_DIR = None -get_default_version = helpers.basic_get_default_version -healthy = helpers.basic_healthy -install_environment = helpers.no_install - - -def run_hook(hook, file_args): - return helpers.run_xargs(hook, helpers.to_cmd(hook), file_args) diff --git a/pre_commit/languages/unsupported.py b/pre_commit/languages/unsupported.py new file mode 100644 index 00000000..f6ad688f --- /dev/null +++ b/pre_commit/languages/unsupported.py @@ -0,0 +1,10 @@ +from __future__ import annotations + +from pre_commit import lang_base + +ENVIRONMENT_DIR = None +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +install_environment = lang_base.no_install +in_env = lang_base.no_env +run_hook = lang_base.basic_run_hook diff --git a/pre_commit/languages/unsupported_script.py b/pre_commit/languages/unsupported_script.py new file mode 100644 index 00000000..1eaa1e27 --- /dev/null +++ b/pre_commit/languages/unsupported_script.py @@ -0,0 +1,32 @@ +from __future__ import annotations + +from collections.abc import Sequence + +from pre_commit import lang_base +from pre_commit.prefix import Prefix + +ENVIRONMENT_DIR = None +get_default_version = lang_base.basic_get_default_version +health_check = lang_base.basic_health_check +install_environment = lang_base.no_install +in_env = lang_base.no_env + + +def run_hook( + prefix: Prefix, + entry: str, + args: Sequence[str], + file_args: Sequence[str], + *, + is_local: bool, + require_serial: bool, + color: bool, +) -> tuple[int, bytes]: + cmd = lang_base.hook_cmd(entry, args) + cmd = (prefix.path(cmd[0]), *cmd[1:]) + return lang_base.run_xargs( + cmd, + file_args, + require_serial=require_serial, + color=color, + ) diff --git a/pre_commit/logging_handler.py b/pre_commit/logging_handler.py index a1e2c086..74772bee 100644 --- a/pre_commit/logging_handler.py +++ b/pre_commit/logging_handler.py @@ -1,12 +1,12 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib import logging +from collections.abc import Generator from pre_commit import color from pre_commit import output - logger = logging.getLogger('pre_commit') LOG_LEVEL_COLORS = { @@ -18,26 +18,22 @@ LOG_LEVEL_COLORS = { class LoggingHandler(logging.Handler): - def __init__(self, use_color): - super(LoggingHandler, self).__init__() + def __init__(self, use_color: bool) -> None: + super().__init__() self.use_color = use_color - def emit(self, record): - output.write_line( - '{} {}'.format( - color.format_color( - '[{}]'.format(record.levelname), - LOG_LEVEL_COLORS[record.levelname], - self.use_color, - ), - record.getMessage(), - ), + def emit(self, record: logging.LogRecord) -> None: + level_msg = color.format_color( + f'[{record.levelname}]', + LOG_LEVEL_COLORS[record.levelname], + self.use_color, ) + output.write_line(f'{level_msg} {record.getMessage()}') @contextlib.contextmanager -def logging_handler(*args, **kwargs): - handler = LoggingHandler(*args, **kwargs) +def logging_handler(use_color: bool) -> Generator[None]: + handler = LoggingHandler(use_color) logger.addHandler(handler) logger.setLevel(logging.INFO) try: diff --git a/pre_commit/main.py b/pre_commit/main.py index dbfbecf6..0c3eefda 100644 --- a/pre_commit/main.py +++ b/pre_commit/main.py @@ -1,17 +1,20 @@ -from __future__ import unicode_literals +from __future__ import annotations import argparse import logging import os import sys +from collections.abc import Sequence import pre_commit.constants as C -from pre_commit import color -from pre_commit import five +from pre_commit import clientlib from pre_commit import git +from pre_commit.color import add_color_option +from pre_commit.commands import hazmat from pre_commit.commands.autoupdate import autoupdate from pre_commit.commands.clean import clean from pre_commit.commands.gc import gc +from pre_commit.commands.hook_impl import hook_impl from pre_commit.commands.init_templatedir import init_templatedir from pre_commit.commands.install_uninstall import install from pre_commit.commands.install_uninstall import install_hooks @@ -20,11 +23,11 @@ from pre_commit.commands.migrate_config import migrate_config from pre_commit.commands.run import run from pre_commit.commands.sample_config import sample_config from pre_commit.commands.try_repo import try_repo +from pre_commit.commands.validate_config import validate_config +from pre_commit.commands.validate_manifest import validate_manifest from pre_commit.error_handler import error_handler -from pre_commit.error_handler import FatalError from pre_commit.logging_handler import logging_handler from pre_commit.store import Store -from pre_commit.util import CalledProcessError logger = logging.getLogger('pre_commit') @@ -35,156 +38,239 @@ logger = logging.getLogger('pre_commit') # pyvenv os.environ.pop('__PYVENV_LAUNCHER__', None) +# https://github.com/getsentry/snuba/pull/5388 +os.environ.pop('PYTHONEXECUTABLE', None) -COMMANDS_NO_GIT = {'clean', 'gc', 'init-templatedir', 'sample-config'} +COMMANDS_NO_GIT = { + 'clean', 'gc', 'hazmat', 'init-templatedir', 'sample-config', + 'validate-config', 'validate-manifest', +} -def _add_color_option(parser): - parser.add_argument( - '--color', default=os.environ.get('PRE_COMMIT_COLOR', 'auto'), - type=color.use_color, - metavar='{' + ','.join(color.COLOR_CHOICES) + '}', - help='Whether to use color in output. Defaults to `%(default)s`.', - ) - - -def _add_config_option(parser): +def _add_config_option(parser: argparse.ArgumentParser) -> None: parser.add_argument( '-c', '--config', default=C.CONFIG_FILE, help='Path to alternate config file', ) -def _add_hook_type_option(parser): +def _add_hook_type_option(parser: argparse.ArgumentParser) -> None: parser.add_argument( - '-t', '--hook-type', choices=( - 'pre-commit', 'pre-push', 'prepare-commit-msg', 'commit-msg', - ), - default='pre-commit', + '-t', '--hook-type', + choices=clientlib.HOOK_TYPES, action='append', dest='hook_types', ) -def _add_run_options(parser): +def _add_run_options(parser: argparse.ArgumentParser) -> None: parser.add_argument('hook', nargs='?', help='A single hook-id to run') - parser.add_argument('--verbose', '-v', action='store_true', default=False) - parser.add_argument( - '--origin', '-o', - help="The origin branch's commit_id when using `git push`.", - ) - parser.add_argument( - '--source', '-s', - help="The remote branch's commit_id when using `git push`.", - ) - parser.add_argument( - '--commit-msg-filename', - help='Filename to check when running during `commit-msg`', - ) - parser.add_argument( - '--hook-stage', choices=C.STAGES, default='commit', - help='The stage during which the hook is fired. One of %(choices)s', - ) - parser.add_argument( - '--show-diff-on-failure', action='store_true', - help='When hooks fail, run `git diff` directly afterward.', - ) + parser.add_argument('--verbose', '-v', action='store_true') mutex_group = parser.add_mutually_exclusive_group(required=False) mutex_group.add_argument( - '--all-files', '-a', action='store_true', default=False, + '--all-files', '-a', action='store_true', help='Run on all the files in the repo.', ) mutex_group.add_argument( '--files', nargs='*', default=[], help='Specific filenames to run hooks on.', ) + parser.add_argument( + '--show-diff-on-failure', action='store_true', + help='When hooks fail, run `git diff` directly afterward.', + ) + parser.add_argument( + '--fail-fast', action='store_true', + help='Stop after the first failing hook.', + ) + parser.add_argument( + '--hook-stage', + choices=clientlib.STAGES, + type=clientlib.transform_stage, + default='pre-commit', + help='The stage during which the hook is fired. One of %(choices)s', + ) + parser.add_argument( + '--remote-branch', help='Remote branch ref used by `git push`.', + ) + parser.add_argument( + '--local-branch', help='Local branch ref used by `git push`.', + ) + parser.add_argument( + '--from-ref', '--source', '-s', + help=( + '(for usage with `--to-ref`) -- this option represents the ' + 'original ref in a `from_ref...to_ref` diff expression. ' + 'For `pre-push` hooks, this represents the branch you are pushing ' + 'to. ' + 'For `post-checkout` hooks, this represents the branch that was ' + 'previously checked out.' + ), + ) + parser.add_argument( + '--to-ref', '--origin', '-o', + help=( + '(for usage with `--from-ref`) -- this option represents the ' + 'destination ref in a `from_ref...to_ref` diff expression. ' + 'For `pre-push` hooks, this represents the branch being pushed. ' + 'For `post-checkout` hooks, this represents the branch that is ' + 'now checked out.' + ), + ) + parser.add_argument( + '--pre-rebase-upstream', help=( + 'The upstream from which the series was forked.' + ), + ) + parser.add_argument( + '--pre-rebase-branch', help=( + 'The branch being rebased, and is not set when ' + 'rebasing the current branch.' + ), + ) + parser.add_argument( + '--commit-msg-filename', + help='Filename to check when running during `commit-msg`', + ) + parser.add_argument( + '--prepare-commit-message-source', + help=( + 'Source of the commit message ' + '(typically the second argument to .git/hooks/prepare-commit-msg)' + ), + ) + parser.add_argument( + '--commit-object-name', + help=( + 'Commit object name ' + '(typically the third argument to .git/hooks/prepare-commit-msg)' + ), + ) + parser.add_argument( + '--remote-name', help='Remote name used by `git push`.', + ) + parser.add_argument('--remote-url', help='Remote url used by `git push`.') + parser.add_argument( + '--checkout-type', + help=( + 'Indicates whether the checkout was a branch checkout ' + '(changing branches, flag=1) or a file checkout (retrieving a ' + 'file from the index, flag=0).' + ), + ) + parser.add_argument( + '--is-squash-merge', + help=( + 'During a post-merge hook, indicates whether the merge was a ' + 'squash merge' + ), + ) + parser.add_argument( + '--rewrite-command', + help=( + 'During a post-rewrite hook, specifies the command that invoked ' + 'the rewrite' + ), + ) -def _adjust_args_and_chdir(args): +def _adjust_args_and_chdir(args: argparse.Namespace) -> None: # `--config` was specified relative to the non-root working directory if os.path.exists(args.config): args.config = os.path.abspath(args.config) if args.command in {'run', 'try-repo'}: args.files = [os.path.abspath(filename) for filename in args.files] + if args.commit_msg_filename is not None: + args.commit_msg_filename = os.path.abspath( + args.commit_msg_filename, + ) if args.command == 'try-repo' and os.path.exists(args.repo): args.repo = os.path.abspath(args.repo) - try: - os.chdir(git.get_root()) - except CalledProcessError: - raise FatalError( - 'git failed. Is it installed, and are you in a Git repository ' - 'directory?', - ) + toplevel = git.get_root() + os.chdir(toplevel) args.config = os.path.relpath(args.config) if args.command in {'run', 'try-repo'}: args.files = [os.path.relpath(filename) for filename in args.files] + if args.commit_msg_filename is not None: + args.commit_msg_filename = os.path.relpath( + args.commit_msg_filename, + ) if args.command == 'try-repo' and os.path.exists(args.repo): args.repo = os.path.relpath(args.repo) -def main(argv=None): +def main(argv: Sequence[str] | None = None) -> int: argv = argv if argv is not None else sys.argv[1:] - argv = [five.to_text(arg) for arg in argv] - parser = argparse.ArgumentParser() + parser = argparse.ArgumentParser(prog='pre-commit') # https://stackoverflow.com/a/8521644/812183 parser.add_argument( '-V', '--version', action='version', - version='%(prog)s {}'.format(C.VERSION), + version=f'%(prog)s {C.VERSION}', ) subparsers = parser.add_subparsers(dest='command') - autoupdate_parser = subparsers.add_parser( + def _add_cmd(name: str, *, help: str) -> argparse.ArgumentParser: + parser = subparsers.add_parser(name, help=help) + add_color_option(parser) + return parser + + autoupdate_parser = _add_cmd( 'autoupdate', help="Auto-update pre-commit config to the latest repos' versions.", ) - _add_color_option(autoupdate_parser) _add_config_option(autoupdate_parser) - autoupdate_parser.add_argument( - '--tags-only', action='store_true', help='LEGACY: for compatibility', - ) autoupdate_parser.add_argument( '--bleeding-edge', action='store_true', help=( - 'Update to the bleeding edge of `master` instead of the latest ' + 'Update to the bleeding edge of `HEAD` instead of the latest ' 'tagged version (the default behavior).' ), ) autoupdate_parser.add_argument( - '--repo', dest='repos', action='append', metavar='REPO', + '--freeze', action='store_true', + help='Store "frozen" hashes in `rev` instead of tag names', + ) + autoupdate_parser.add_argument( + '--repo', dest='repos', action='append', metavar='REPO', default=[], help='Only update this repository -- may be specified multiple times.', ) - - clean_parser = subparsers.add_parser( - 'clean', help='Clean out pre-commit files.', + autoupdate_parser.add_argument( + '-j', '--jobs', type=int, default=1, + help='Number of threads to use. (default %(default)s).', ) - _add_color_option(clean_parser) - _add_config_option(clean_parser) - gc_parser = subparsers.add_parser('gc', help='Clean unused cached repos.') - _add_color_option(gc_parser) - _add_config_option(gc_parser) + _add_cmd('clean', help='Clean out pre-commit files.') - init_templatedir_parser = subparsers.add_parser( + _add_cmd('gc', help='Clean unused cached repos.') + + hazmat_parser = _add_cmd( + 'hazmat', help='Composable tools for rare use in hook `entry`.', + ) + hazmat.add_parsers(hazmat_parser) + + init_templatedir_parser = _add_cmd( 'init-templatedir', help=( 'Install hook script in a directory intended for use with ' '`git config init.templateDir`.' ), ) - _add_color_option(init_templatedir_parser) _add_config_option(init_templatedir_parser) init_templatedir_parser.add_argument( 'directory', help='The directory in which to write the hook script.', ) + init_templatedir_parser.add_argument( + '--no-allow-missing-config', + action='store_false', + dest='allow_missing_config', + help='Assume cloned repos should have a `pre-commit` config.', + ) _add_hook_type_option(init_templatedir_parser) - install_parser = subparsers.add_parser( - 'install', help='Install the pre-commit script.', - ) - _add_color_option(install_parser) + install_parser = _add_cmd('install', help='Install the pre-commit script.') _add_config_option(install_parser) install_parser.add_argument( '-f', '--overwrite', action='store_true', @@ -199,14 +285,14 @@ def main(argv=None): ) _add_hook_type_option(install_parser) install_parser.add_argument( - '--allow-missing-config', action='store_true', default=False, + '--allow-missing-config', action='store_true', help=( 'Whether to allow a missing `pre-commit` configuration file ' 'or exit with a failure code.' ), ) - install_hooks_parser = subparsers.add_parser( + install_hooks_parser = _add_cmd( 'install-hooks', help=( 'Install hook environments for all environments in the config ' @@ -214,32 +300,24 @@ def main(argv=None): 'useful.' ), ) - _add_color_option(install_hooks_parser) _add_config_option(install_hooks_parser) - migrate_config_parser = subparsers.add_parser( + migrate_config_parser = _add_cmd( 'migrate-config', help='Migrate list configuration to new map configuration.', ) - _add_color_option(migrate_config_parser) _add_config_option(migrate_config_parser) - run_parser = subparsers.add_parser('run', help='Run hooks.') - _add_color_option(run_parser) + run_parser = _add_cmd('run', help='Run hooks.') _add_config_option(run_parser) _add_run_options(run_parser) - sample_config_parser = subparsers.add_parser( - 'sample-config', help='Produce a sample {} file'.format(C.CONFIG_FILE), - ) - _add_color_option(sample_config_parser) - _add_config_option(sample_config_parser) + _add_cmd('sample-config', help=f'Produce a sample {C.CONFIG_FILE} file') - try_repo_parser = subparsers.add_parser( + try_repo_parser = _add_cmd( 'try-repo', help='Try the hooks in a repository, useful for developing new hooks.', ) - _add_color_option(try_repo_parser) _add_config_option(try_repo_parser) try_repo_parser.add_argument( 'repo', help='Repository to source hooks from.', @@ -253,18 +331,39 @@ def main(argv=None): ) _add_run_options(try_repo_parser) - uninstall_parser = subparsers.add_parser( + uninstall_parser = _add_cmd( 'uninstall', help='Uninstall the pre-commit script.', ) - _add_color_option(uninstall_parser) _add_config_option(uninstall_parser) _add_hook_type_option(uninstall_parser) + validate_config_parser = _add_cmd( + 'validate-config', help='Validate .pre-commit-config.yaml files', + ) + validate_config_parser.add_argument('filenames', nargs='*') + + validate_manifest_parser = _add_cmd( + 'validate-manifest', help='Validate .pre-commit-hooks.yaml files', + ) + validate_manifest_parser.add_argument('filenames', nargs='*') + + # does not use `_add_cmd` because it doesn't use `--color` help = subparsers.add_parser( 'help', help='Show help for a specific command.', ) help.add_argument('help_cmd', nargs='?', help='Command to show help for.') + # not intended for users to call this directly + hook_impl_parser = subparsers.add_parser('hook-impl') + add_color_option(hook_impl_parser) + _add_config_option(hook_impl_parser) + hook_impl_parser.add_argument('--hook-type') + hook_impl_parser.add_argument('--hook-dir') + hook_impl_parser.add_argument( + '--skip-on-missing-config', action='store_true', + ) + hook_impl_parser.add_argument(dest='rest', nargs=argparse.REMAINDER) + # argparse doesn't really provide a way to use a `default` subparser if len(argv) == 0: argv = ['run'] @@ -276,37 +375,51 @@ def main(argv=None): parser.parse_args(['--help']) with error_handler(), logging_handler(args.color): - if args.command not in COMMANDS_NO_GIT: - _adjust_args_and_chdir(args) - git.check_for_cygwin_mismatch() store = Store() - store.mark_config_used(args.config) + + if args.command not in COMMANDS_NO_GIT: + _adjust_args_and_chdir(args) + store.mark_config_used(args.config) if args.command == 'autoupdate': - if args.tags_only: - logger.warning('--tags-only is the default') return autoupdate( - args.config, store, + args.config, tags_only=not args.bleeding_edge, + freeze=args.freeze, repos=args.repos, + jobs=args.jobs, ) elif args.command == 'clean': return clean(store) elif args.command == 'gc': return gc(store) + elif args.command == 'hazmat': + return hazmat.impl(args) + elif args.command == 'hook-impl': + return hook_impl( + store, + config=args.config, + color=args.color, + hook_type=args.hook_type, + hook_dir=args.hook_dir, + skip_on_missing_config=args.skip_on_missing_config, + args=args.rest[1:], + ) elif args.command == 'install': return install( args.config, store, - overwrite=args.overwrite, hooks=args.install_hooks, - hook_type=args.hook_type, + hook_types=args.hook_types, + overwrite=args.overwrite, + hooks=args.install_hooks, skip_on_missing_config=args.allow_missing_config, ) elif args.command == 'init-templatedir': return init_templatedir( - args.config, store, - args.directory, hook_type=args.hook_type, + args.config, store, args.directory, + hook_types=args.hook_types, + skip_on_missing_config=args.allow_missing_config, ) elif args.command == 'install-hooks': return install_hooks(args.config, store) @@ -319,16 +432,23 @@ def main(argv=None): elif args.command == 'try-repo': return try_repo(args) elif args.command == 'uninstall': - return uninstall(hook_type=args.hook_type) + return uninstall( + config_file=args.config, + hook_types=args.hook_types, + ) + elif args.command == 'validate-config': + return validate_config(args.filenames) + elif args.command == 'validate-manifest': + return validate_manifest(args.filenames) else: raise NotImplementedError( - 'Command {} not implemented.'.format(args.command), + f'Command {args.command} not implemented.', ) raise AssertionError( - 'Command {} failed to exit with a returncode'.format(args.command), + f'Command {args.command} failed to exit with a returncode', ) if __name__ == '__main__': - exit(main()) + raise SystemExit(main()) diff --git a/pre_commit/make_archives.py b/pre_commit/make_archives.py deleted file mode 100644 index 9dd9e5e7..00000000 --- a/pre_commit/make_archives.py +++ /dev/null @@ -1,68 +0,0 @@ -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - -import argparse -import os.path -import tarfile - -from pre_commit import output -from pre_commit.util import cmd_output -from pre_commit.util import rmtree -from pre_commit.util import tmpdir - - -# This is a script for generating the tarred resources for git repo -# dependencies. Currently it's just for "vendoring" ruby support packages. - - -REPOS = ( - ('rbenv', 'git://github.com/rbenv/rbenv', 'e60ad4a'), - ('ruby-build', 'git://github.com/rbenv/ruby-build', '9bc9971'), - ( - 'ruby-download', - 'git://github.com/garnieretienne/rvm-download', - '09bd7c6', - ), -) - - -def make_archive(name, repo, ref, destdir): - """Makes an archive of a repository in the given destdir. - - :param text name: Name to give the archive. For instance foo. The file - that is created will be called foo.tar.gz. - :param text repo: Repository to clone. - :param text ref: Tag/SHA/branch to check out. - :param text destdir: Directory to place archives in. - """ - output_path = os.path.join(destdir, name + '.tar.gz') - with tmpdir() as tempdir: - # Clone the repository to the temporary directory - cmd_output('git', 'clone', repo, tempdir) - cmd_output('git', 'checkout', ref, cwd=tempdir) - - # We don't want the '.git' directory - # It adds a bunch of size to the archive and we don't use it at - # runtime - rmtree(os.path.join(tempdir, '.git')) - - with tarfile.open(output_path, 'w|gz') as tf: - tf.add(tempdir, name) - - return output_path - - -def main(argv=None): - parser = argparse.ArgumentParser() - parser.add_argument('--dest', default='pre_commit/resources') - args = parser.parse_args(argv) - for archive_name, repo, ref in REPOS: - output.write_line( - 'Making {}.tar.gz for {}@{}'.format(archive_name, repo, ref), - ) - make_archive(archive_name, repo, ref, args.dest) - - -if __name__ == '__main__': - exit(main()) diff --git a/pre_commit/meta_hooks/check_hooks_apply.py b/pre_commit/meta_hooks/check_hooks_apply.py index b1ccdac3..84c142b4 100644 --- a/pre_commit/meta_hooks/check_hooks_apply.py +++ b/pre_commit/meta_hooks/check_hooks_apply.py @@ -1,4 +1,7 @@ +from __future__ import annotations + import argparse +from collections.abc import Sequence import pre_commit.constants as C from pre_commit import git @@ -8,21 +11,24 @@ from pre_commit.repository import all_hooks from pre_commit.store import Store -def check_all_hooks_match_files(config_file): - classifier = Classifier(git.get_all_files()) +def check_all_hooks_match_files(config_file: str) -> int: + config = load_config(config_file) + classifier = Classifier.from_config( + git.get_all_files(), config['files'], config['exclude'], + ) retv = 0 - for hook in all_hooks(load_config(config_file), Store()): + for hook in all_hooks(config, Store()): if hook.always_run or hook.language == 'fail': continue - elif not classifier.filenames_for_hook(hook): - print('{} does not apply to this repository'.format(hook.id)) + elif not any(classifier.filenames_for_hook(hook)): + print(f'{hook.id} does not apply to this repository') retv = 1 return retv -def main(argv=None): +def main(argv: Sequence[str] | None = None) -> int: parser = argparse.ArgumentParser() parser.add_argument('filenames', nargs='*', default=[C.CONFIG_FILE]) args = parser.parse_args(argv) @@ -34,4 +40,4 @@ def main(argv=None): if __name__ == '__main__': - exit(main()) + raise SystemExit(main()) diff --git a/pre_commit/meta_hooks/check_useless_excludes.py b/pre_commit/meta_hooks/check_useless_excludes.py index c4860db3..664251a4 100644 --- a/pre_commit/meta_hooks/check_useless_excludes.py +++ b/pre_commit/meta_hooks/check_useless_excludes.py @@ -1,7 +1,9 @@ -from __future__ import print_function +from __future__ import annotations import argparse import re +from collections.abc import Iterable +from collections.abc import Sequence from cfgv import apply_defaults @@ -12,7 +14,11 @@ from pre_commit.clientlib import MANIFEST_HOOK_DICT from pre_commit.commands.run import Classifier -def exclude_matches_any(filenames, include, exclude): +def exclude_matches_any( + filenames: Iterable[str], + include: str, + exclude: str, +) -> bool: if exclude == '^$': return True include_re, exclude_re = re.compile(include), re.compile(exclude) @@ -22,39 +28,47 @@ def exclude_matches_any(filenames, include, exclude): return False -def check_useless_excludes(config_file): +def check_useless_excludes(config_file: str) -> int: config = load_config(config_file) - classifier = Classifier(git.get_all_files()) + filenames = git.get_all_files() + classifier = Classifier.from_config( + filenames, config['files'], config['exclude'], + ) retv = 0 exclude = config['exclude'] - if not exclude_matches_any(classifier.filenames, '', exclude): + if not exclude_matches_any(filenames, '', exclude): print( - 'The global exclude pattern {!r} does not match any files' - .format(exclude), + f'The global exclude pattern {exclude!r} does not match any files', ) retv = 1 for repo in config['repos']: for hook in repo['hooks']: + # the default of manifest hooks is `types: [file]` but we may + # be configuring a symlink hook while there's a broken symlink + hook.setdefault('types', []) # Not actually a manifest dict, but this more accurately reflects # the defaults applied during runtime hook = apply_defaults(hook, MANIFEST_HOOK_DICT) - names = classifier.filenames - types, exclude_types = hook['types'], hook['exclude_types'] - names = classifier.by_types(names, types, exclude_types) + names = classifier.by_types( + classifier.filenames, + hook['types'], + hook['types_or'], + hook['exclude_types'], + ) include, exclude = hook['files'], hook['exclude'] if not exclude_matches_any(names, include, exclude): print( - 'The exclude pattern {!r} for {} does not match any files' - .format(exclude, hook['id']), + f'The exclude pattern {exclude!r} for {hook["id"]} does ' + f'not match any files', ) retv = 1 return retv -def main(argv=None): +def main(argv: Sequence[str] | None = None) -> int: parser = argparse.ArgumentParser() parser.add_argument('filenames', nargs='*', default=[C.CONFIG_FILE]) args = parser.parse_args(argv) @@ -66,4 +80,4 @@ def main(argv=None): if __name__ == '__main__': - exit(main()) + raise SystemExit(main()) diff --git a/pre_commit/meta_hooks/identity.py b/pre_commit/meta_hooks/identity.py index ae7377b8..3e20bbc6 100644 --- a/pre_commit/meta_hooks/identity.py +++ b/pre_commit/meta_hooks/identity.py @@ -1,13 +1,17 @@ +from __future__ import annotations + import sys +from collections.abc import Sequence from pre_commit import output -def main(argv=None): +def main(argv: Sequence[str] | None = None) -> int: argv = argv if argv is not None else sys.argv[1:] for arg in argv: output.write_line(arg) + return 0 if __name__ == '__main__': - exit(main()) + raise SystemExit(main()) diff --git a/pre_commit/output.py b/pre_commit/output.py index 478ad5e6..4bcf27f9 100644 --- a/pre_commit/output.py +++ b/pre_commit/output.py @@ -1,88 +1,33 @@ -from __future__ import unicode_literals +from __future__ import annotations +import contextlib import sys - -from pre_commit import color -from pre_commit import five -from pre_commit.util import noop_context +from typing import Any +from typing import IO -def get_hook_message( - start, - postfix='', - end_msg=None, - end_len=0, - end_color=None, - use_color=None, - cols=80, -): - """Prints a message for running a hook. - - This currently supports three approaches: - - # Print `start` followed by dots, leaving 6 characters at the end - >>> print_hook_message('start', end_len=6) - start............................................................... - - # Print `start` followed by dots with the end message colored if coloring - # is specified and a newline afterwards - >>> print_hook_message( - 'start', - end_msg='end', - end_color=color.RED, - use_color=True, - ) - start...................................................................end - - # Print `start` followed by dots, followed by the `postfix` message - # uncolored, followed by the `end_msg` colored if specified and a newline - # afterwards - >>> print_hook_message( - 'start', - postfix='postfix ', - end_msg='end', - end_color=color.RED, - use_color=True, - ) - start...........................................................postfix end - """ - if bool(end_msg) == bool(end_len): - raise ValueError('Expected one of (`end_msg`, `end_len`)') - if end_msg is not None and (end_color is None or use_color is None): - raise ValueError( - '`end_color` and `use_color` are required with `end_msg`', - ) - - if end_len: - return start + '.' * (cols - len(start) - end_len - 1) - else: - return '{}{}{}{}\n'.format( - start, - '.' * (cols - len(start) - len(postfix) - len(end_msg) - 1), - postfix, - color.format_color(end_msg, end_color, use_color), - ) - - -stdout_byte_stream = getattr(sys.stdout, 'buffer', sys.stdout) - - -def write(s, stream=stdout_byte_stream): - stream.write(five.to_bytes(s)) +def write(s: str, stream: IO[bytes] = sys.stdout.buffer) -> None: + stream.write(s.encode()) stream.flush() -def write_line(s=None, stream=stdout_byte_stream, logfile_name=None): - output_streams = [stream] - if logfile_name: - ctx = open(logfile_name, 'ab') - output_streams.append(ctx) - else: - ctx = noop_context() +def write_line_b( + s: bytes | None = None, + stream: IO[bytes] = sys.stdout.buffer, + logfile_name: str | None = None, +) -> None: + with contextlib.ExitStack() as exit_stack: + output_streams = [stream] + if logfile_name: + stream = exit_stack.enter_context(open(logfile_name, 'ab')) + output_streams.append(stream) - with ctx: for output_stream in output_streams: if s is not None: - output_stream.write(five.to_bytes(s)) + output_stream.write(s) output_stream.write(b'\n') output_stream.flush() + + +def write_line(s: str | None = None, **kwargs: Any) -> None: + write_line_b(s.encode() if s is not None else s, **kwargs) diff --git a/pre_commit/parse_shebang.py b/pre_commit/parse_shebang.py index 5a2ba72f..043a9b5d 100644 --- a/pre_commit/parse_shebang.py +++ b/pre_commit/parse_shebang.py @@ -1,35 +1,36 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os.path +from collections.abc import Mapping +from typing import NoReturn from identify.identify import parse_shebang_from_file class ExecutableNotFoundError(OSError): - def to_output(self): - return (1, self.args[0].encode('UTF-8'), b'') + def to_output(self) -> tuple[int, bytes, None]: + return (1, self.args[0].encode(), None) -def parse_filename(filename): +def parse_filename(filename: str) -> tuple[str, ...]: if not os.path.exists(filename): return () else: return parse_shebang_from_file(filename) -def find_executable(exe, _environ=None): +def find_executable( + exe: str, *, env: Mapping[str, str] | None = None, +) -> str | None: exe = os.path.normpath(exe) if os.sep in exe: return exe - environ = _environ if _environ is not None else os.environ + environ = env if env is not None else os.environ if 'PATHEXT' in environ: - possible_exe_names = tuple( - exe + ext.lower() for ext in environ['PATHEXT'].split(os.pathsep) - ) + (exe,) - + exts = environ['PATHEXT'].split(os.pathsep) + possible_exe_names = tuple(f'{exe}{ext}' for ext in exts) + (exe,) else: possible_exe_names = (exe,) @@ -42,24 +43,30 @@ def find_executable(exe, _environ=None): return None -def normexe(orig): - def _error(msg): - raise ExecutableNotFoundError('Executable `{}` {}'.format(orig, msg)) +def normexe(orig: str, *, env: Mapping[str, str] | None = None) -> str: + def _error(msg: str) -> NoReturn: + raise ExecutableNotFoundError(f'Executable `{orig}` {msg}') if os.sep not in orig and (not os.altsep or os.altsep not in orig): - exe = find_executable(orig) + exe = find_executable(orig, env=env) if exe is None: _error('not found') return exe - elif not os.access(orig, os.X_OK): - _error('not found') elif os.path.isdir(orig): _error('is a directory') + elif not os.path.isfile(orig): + _error('not found') + elif not os.access(orig, os.X_OK): # pragma: win32 no cover + _error('is not executable') else: return orig -def normalize_cmd(cmd): +def normalize_cmd( + cmd: tuple[str, ...], + *, + env: Mapping[str, str] | None = None, +) -> tuple[str, ...]: """Fixes for the following issues on windows - https://bugs.python.org/issue8557 - windows does not parse shebangs @@ -67,12 +74,12 @@ def normalize_cmd(cmd): This function also makes deep-path shebangs work just fine """ # Use PATH to determine the executable - exe = normexe(cmd[0]) + exe = normexe(cmd[0], env=env) # Figure out the shebang from the resulting command cmd = parse_filename(exe) + (exe,) + cmd[1:] # This could have given us back another bare executable - exe = normexe(cmd[0]) + exe = normexe(cmd[0], env=env) return (exe,) + cmd[1:] diff --git a/pre_commit/prefix.py b/pre_commit/prefix.py index f8a8a9d6..f1b28c1d 100644 --- a/pre_commit/prefix.py +++ b/pre_commit/prefix.py @@ -1,18 +1,18 @@ -from __future__ import unicode_literals +from __future__ import annotations -import collections import os.path +from typing import NamedTuple -class Prefix(collections.namedtuple('Prefix', ('prefix_dir',))): - __slots__ = () +class Prefix(NamedTuple): + prefix_dir: str - def path(self, *parts): + def path(self, *parts: str) -> str: return os.path.normpath(os.path.join(self.prefix_dir, *parts)) - def exists(self, *parts): + def exists(self, *parts: str) -> bool: return os.path.exists(self.path(*parts)) - def star(self, end): + def star(self, end: str) -> tuple[str, ...]: paths = os.listdir(self.prefix_dir) return tuple(path for path in paths if path.endswith(end)) diff --git a/pre_commit/repository.py b/pre_commit/repository.py index 5b12a98c..a9461ab6 100644 --- a/pre_commit/repository.py +++ b/pre_commit/repository.py @@ -1,134 +1,119 @@ -from __future__ import unicode_literals +from __future__ import annotations -import collections -import io import json import logging import os +from collections.abc import Sequence +from typing import Any import pre_commit.constants as C -from pre_commit import five +from pre_commit.all_languages import languages from pre_commit.clientlib import load_manifest from pre_commit.clientlib import LOCAL -from pre_commit.clientlib import MANIFEST_HOOK_DICT from pre_commit.clientlib import META -from pre_commit.languages.all import languages -from pre_commit.languages.helpers import environment_dir +from pre_commit.hook import Hook +from pre_commit.lang_base import environment_dir from pre_commit.prefix import Prefix -from pre_commit.util import parse_version +from pre_commit.store import Store +from pre_commit.util import clean_path_on_failure from pre_commit.util import rmtree logger = logging.getLogger('pre_commit') -def _state(additional_deps): - return {'additional_dependencies': sorted(additional_deps)} +def _state_filename_v1(venv: str) -> str: + return os.path.join(venv, '.install_state_v1') -def _state_filename(prefix, venv): - return prefix.path(venv, '.install_state_v' + C.INSTALLED_STATE_VERSION) +def _state_filename_v2(venv: str) -> str: + return os.path.join(venv, '.install_state_v2') -def _read_state(prefix, venv): - filename = _state_filename(prefix, venv) +def _state(additional_deps: Sequence[str]) -> object: + return {'additional_dependencies': additional_deps} + + +def _read_state(venv: str) -> object | None: + filename = _state_filename_v1(venv) if not os.path.exists(filename): return None else: - with io.open(filename) as f: + with open(filename) as f: return json.load(f) -def _write_state(prefix, venv, state): - state_filename = _state_filename(prefix, venv) - staging = state_filename + 'staging' - with io.open(staging, 'w') as state_file: - state_file.write(five.to_text(json.dumps(state))) - # Move the file into place atomically to indicate we've installed - os.rename(staging, state_filename) +def _hook_installed(hook: Hook) -> bool: + lang = languages[hook.language] + if lang.ENVIRONMENT_DIR is None: + return True + + venv = environment_dir( + hook.prefix, + lang.ENVIRONMENT_DIR, + hook.language_version, + ) + return ( + ( + os.path.exists(_state_filename_v2(venv)) or + _read_state(venv) == _state(hook.additional_dependencies) + ) and + not lang.health_check(hook.prefix, hook.language_version) + ) -_KEYS = tuple(item.key for item in MANIFEST_HOOK_DICT.items) +def _hook_install(hook: Hook) -> None: + logger.info(f'Installing environment for {hook.src}.') + logger.info('Once installed this environment will be reused.') + logger.info('This may take a few minutes...') + lang = languages[hook.language] + assert lang.ENVIRONMENT_DIR is not None -class Hook(collections.namedtuple('Hook', ('src', 'prefix') + _KEYS)): - __slots__ = () + venv = environment_dir( + hook.prefix, + lang.ENVIRONMENT_DIR, + hook.language_version, + ) - @property - def install_key(self): - return ( - self.prefix, - self.language, - self.language_version, - tuple(self.additional_dependencies), - ) - - def installed(self): - lang = languages[self.language] - venv = environment_dir(lang.ENVIRONMENT_DIR, self.language_version) - return ( - venv is None or ( - ( - _read_state(self.prefix, venv) == - _state(self.additional_dependencies) - ) and - lang.healthy(self.prefix, self.language_version) - ) - ) - - def install(self): - logger.info('Installing environment for {}.'.format(self.src)) - logger.info('Once installed this environment will be reused.') - logger.info('This may take a few minutes...') - - lang = languages[self.language] - venv = environment_dir(lang.ENVIRONMENT_DIR, self.language_version) - - # There's potentially incomplete cleanup from previous runs - # Clean it up! - if self.prefix.exists(venv): - rmtree(self.prefix.path(venv)) + # There's potentially incomplete cleanup from previous runs + # Clean it up! + if os.path.exists(venv): + rmtree(venv) + with clean_path_on_failure(venv): lang.install_environment( - self.prefix, self.language_version, self.additional_dependencies, + hook.prefix, hook.language_version, hook.additional_dependencies, ) - # Write our state to indicate we're installed - _write_state(self.prefix, venv, _state(self.additional_dependencies)) - - def run(self, file_args): - lang = languages[self.language] - return lang.run_hook(self, file_args) - - @classmethod - def create(cls, src, prefix, dct): - # TODO: have cfgv do this (?) - extra_keys = set(dct) - set(_KEYS) - if extra_keys: - logger.warning( - 'Unexpected key(s) present on {} => {}: ' - '{}'.format(src, dct['id'], ', '.join(sorted(extra_keys))), + health_error = lang.health_check(hook.prefix, hook.language_version) + if health_error: + raise AssertionError( + f'BUG: expected environment for {hook.language} to be healthy ' + f'immediately after install, please open an issue describing ' + f'your environment\n\n' + f'more info:\n\n{health_error}', ) - return cls(src=src, prefix=prefix, **{k: dct[k] for k in _KEYS}) + + # TODO: remove v1 state writing, no longer needed after pre-commit 3.0 + # Write our state to indicate we're installed + state_filename = _state_filename_v1(venv) + staging = f'{state_filename}staging' + with open(staging, 'w') as state_file: + state_file.write(json.dumps(_state(hook.additional_dependencies))) + # Move the file into place atomically to indicate we've installed + os.replace(staging, state_filename) + + open(_state_filename_v2(venv), 'a+').close() -def _hook(*hook_dicts, **kwargs): - root_config = kwargs.pop('root_config') - assert not kwargs, kwargs +def _hook( + *hook_dicts: dict[str, Any], + root_config: dict[str, Any], +) -> dict[str, Any]: ret, rest = dict(hook_dicts[0]), hook_dicts[1:] for dct in rest: ret.update(dct) - version = ret['minimum_pre_commit_version'] - if parse_version(version) > parse_version(C.VERSION): - logger.error( - 'The hook `{}` requires pre-commit version {} but version {} ' - 'is installed. ' - 'Perhaps run `pip install --upgrade pre-commit`.'.format( - ret['id'], version, C.VERSION, - ), - ) - exit(1) - lang = ret['language'] if ret['language_version'] == C.DEFAULT: ret['language_version'] = root_config['default_language_version'][lang] @@ -138,13 +123,35 @@ def _hook(*hook_dicts, **kwargs): if not ret['stages']: ret['stages'] = root_config['default_stages'] + if languages[lang].ENVIRONMENT_DIR is None: + if ret['language_version'] != C.DEFAULT: + logger.error( + f'The hook `{ret["id"]}` specifies `language_version` but is ' + f'using language `{lang}` which does not install an ' + f'environment. ' + f'Perhaps you meant to use a specific language?', + ) + exit(1) + if ret['additional_dependencies']: + logger.error( + f'The hook `{ret["id"]}` specifies `additional_dependencies` ' + f'but is using language `{lang}` which does not install an ' + f'environment. ' + f'Perhaps you meant to use a specific language?', + ) + exit(1) + return ret -def _non_cloned_repository_hooks(repo_config, store, root_config): - def _prefix(language_name, deps): +def _non_cloned_repository_hooks( + repo_config: dict[str, Any], + store: Store, + root_config: dict[str, Any], +) -> tuple[Hook, ...]: + def _prefix(language_name: str, deps: Sequence[str]) -> Prefix: language = languages[language_name] - # pcre / pygrep / script / system / docker_image do not have + # pygrep / script / system / docker_image do not have # environments so they work out of the current directory if language.ENVIRONMENT_DIR is None: return Prefix(os.getcwd()) @@ -161,7 +168,11 @@ def _non_cloned_repository_hooks(repo_config, store, root_config): ) -def _cloned_repository_hooks(repo_config, store, root_config): +def _cloned_repository_hooks( + repo_config: dict[str, Any], + store: Store, + root_config: dict[str, Any], +) -> tuple[Hook, ...]: repo, rev = repo_config['repo'], repo_config['rev'] manifest_path = os.path.join(store.clone(repo, rev), C.MANIFEST_FILE) by_id = {hook['id']: hook for hook in load_manifest(manifest_path)} @@ -169,10 +180,9 @@ def _cloned_repository_hooks(repo_config, store, root_config): for hook in repo_config['hooks']: if hook['id'] not in by_id: logger.error( - '`{}` is not present in repository {}. ' - 'Typo? Perhaps it is introduced in a newer version? ' - 'Often `pre-commit autoupdate` fixes this.' - .format(hook['id'], repo), + f'`{hook["id"]}` is not present in repository {repo}. ' + f'Typo? Perhaps it is introduced in a newer version? ' + f'Often `pre-commit autoupdate` fixes this.', ) exit(1) @@ -190,19 +200,23 @@ def _cloned_repository_hooks(repo_config, store, root_config): ) -def _repository_hooks(repo_config, store, root_config): +def _repository_hooks( + repo_config: dict[str, Any], + store: Store, + root_config: dict[str, Any], +) -> tuple[Hook, ...]: if repo_config['repo'] in {LOCAL, META}: return _non_cloned_repository_hooks(repo_config, store, root_config) else: return _cloned_repository_hooks(repo_config, store, root_config) -def install_hook_envs(hooks, store): - def _need_installed(): - seen = set() +def install_hook_envs(hooks: Sequence[Hook], store: Store) -> None: + def _need_installed() -> list[Hook]: + seen: set[tuple[Prefix, str, str, tuple[str, ...]]] = set() ret = [] for hook in hooks: - if hook.install_key not in seen and not hook.installed(): + if hook.install_key not in seen and not _hook_installed(hook): ret.append(hook) seen.add(hook.install_key) return ret @@ -212,10 +226,10 @@ def install_hook_envs(hooks, store): with store.exclusive_lock(): # Another process may have already completed this work for hook in _need_installed(): - hook.install() + _hook_install(hook) -def all_hooks(root_config, store): +def all_hooks(root_config: dict[str, Any], store: Store) -> tuple[Hook, ...]: return tuple( hook for repo in root_config['repos'] diff --git a/pre_commit/resources/empty_template_LICENSE.renv b/pre_commit/resources/empty_template_LICENSE.renv new file mode 100644 index 00000000..253c5d1a --- /dev/null +++ b/pre_commit/resources/empty_template_LICENSE.renv @@ -0,0 +1,7 @@ +Copyright 2021 RStudio, PBC + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/pre_commit/resources/empty_template_Makefile.PL b/pre_commit/resources/empty_template_Makefile.PL new file mode 100644 index 00000000..45a0ba37 --- /dev/null +++ b/pre_commit/resources/empty_template_Makefile.PL @@ -0,0 +1,6 @@ +use ExtUtils::MakeMaker; + +WriteMakefile( + NAME => "PreCommitPlaceholder", + VERSION => "0.0.1", +); diff --git a/pre_commit/resources/empty_template_activate.R b/pre_commit/resources/empty_template_activate.R new file mode 100644 index 00000000..d8d092cc --- /dev/null +++ b/pre_commit/resources/empty_template_activate.R @@ -0,0 +1,440 @@ + +local({ + + # the requested version of renv + version <- "0.12.5" + + # the project directory + project <- getwd() + + # avoid recursion + if (!is.na(Sys.getenv("RENV_R_INITIALIZING", unset = NA))) + return(invisible(TRUE)) + + # signal that we're loading renv during R startup + Sys.setenv("RENV_R_INITIALIZING" = "true") + on.exit(Sys.unsetenv("RENV_R_INITIALIZING"), add = TRUE) + + # signal that we've consented to use renv + options(renv.consent = TRUE) + + # load the 'utils' package eagerly -- this ensures that renv shims, which + # mask 'utils' packages, will come first on the search path + library(utils, lib.loc = .Library) + + # check to see if renv has already been loaded + if ("renv" %in% loadedNamespaces()) { + + # if renv has already been loaded, and it's the requested version of renv, + # nothing to do + spec <- .getNamespaceInfo(.getNamespace("renv"), "spec") + if (identical(spec[["version"]], version)) + return(invisible(TRUE)) + + # otherwise, unload and attempt to load the correct version of renv + unloadNamespace("renv") + + } + + # load bootstrap tools + bootstrap <- function(version, library) { + + # attempt to download renv + tarball <- tryCatch(renv_bootstrap_download(version), error = identity) + if (inherits(tarball, "error")) + stop("failed to download renv ", version) + + # now attempt to install + status <- tryCatch(renv_bootstrap_install(version, tarball, library), error = identity) + if (inherits(status, "error")) + stop("failed to install renv ", version) + + } + + renv_bootstrap_tests_running <- function() { + getOption("renv.tests.running", default = FALSE) + } + + renv_bootstrap_repos <- function() { + + # check for repos override + repos <- Sys.getenv("RENV_CONFIG_REPOS_OVERRIDE", unset = NA) + if (!is.na(repos)) + return(repos) + + # if we're testing, re-use the test repositories + if (renv_bootstrap_tests_running()) + return(getOption("renv.tests.repos")) + + # retrieve current repos + repos <- getOption("repos") + + # ensure @CRAN@ entries are resolved + repos[repos == "@CRAN@"] <- "https://cloud.r-project.org" + + # add in renv.bootstrap.repos if set + default <- c(CRAN = "https://cloud.r-project.org") + extra <- getOption("renv.bootstrap.repos", default = default) + repos <- c(repos, extra) + + # remove duplicates that might've snuck in + dupes <- duplicated(repos) | duplicated(names(repos)) + repos[!dupes] + + } + + renv_bootstrap_download <- function(version) { + + # if the renv version number has 4 components, assume it must + # be retrieved via github + nv <- numeric_version(version) + components <- unclass(nv)[[1]] + + methods <- if (length(components) == 4L) { + list( + renv_bootstrap_download_github + ) + } else { + list( + renv_bootstrap_download_cran_latest, + renv_bootstrap_download_cran_archive + ) + } + + for (method in methods) { + path <- tryCatch(method(version), error = identity) + if (is.character(path) && file.exists(path)) + return(path) + } + + stop("failed to download renv ", version) + + } + + renv_bootstrap_download_impl <- function(url, destfile) { + + mode <- "wb" + + # https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17715 + fixup <- + Sys.info()[["sysname"]] == "Windows" && + substring(url, 1L, 5L) == "file:" + + if (fixup) + mode <- "w+b" + + utils::download.file( + url = url, + destfile = destfile, + mode = mode, + quiet = TRUE + ) + + } + + renv_bootstrap_download_cran_latest <- function(version) { + + repos <- renv_bootstrap_download_cran_latest_find(version) + + message("* Downloading renv ", version, " from CRAN ... ", appendLF = FALSE) + + info <- tryCatch( + utils::download.packages( + pkgs = "renv", + repos = repos, + destdir = tempdir(), + quiet = TRUE + ), + condition = identity + ) + + if (inherits(info, "condition")) { + message("FAILED") + return(FALSE) + } + + message("OK") + info[1, 2] + + } + + renv_bootstrap_download_cran_latest_find <- function(version) { + + all <- renv_bootstrap_repos() + + for (repos in all) { + + db <- tryCatch( + as.data.frame( + x = utils::available.packages(repos = repos), + stringsAsFactors = FALSE + ), + error = identity + ) + + if (inherits(db, "error")) + next + + entry <- db[db$Package %in% "renv" & db$Version %in% version, ] + if (nrow(entry) == 0) + next + + return(repos) + + } + + fmt <- "renv %s is not available from your declared package repositories" + stop(sprintf(fmt, version)) + + } + + renv_bootstrap_download_cran_archive <- function(version) { + + name <- sprintf("renv_%s.tar.gz", version) + repos <- renv_bootstrap_repos() + urls <- file.path(repos, "src/contrib/Archive/renv", name) + destfile <- file.path(tempdir(), name) + + message("* Downloading renv ", version, " from CRAN archive ... ", appendLF = FALSE) + + for (url in urls) { + + status <- tryCatch( + renv_bootstrap_download_impl(url, destfile), + condition = identity + ) + + if (identical(status, 0L)) { + message("OK") + return(destfile) + } + + } + + message("FAILED") + return(FALSE) + + } + + renv_bootstrap_download_github <- function(version) { + + enabled <- Sys.getenv("RENV_BOOTSTRAP_FROM_GITHUB", unset = "TRUE") + if (!identical(enabled, "TRUE")) + return(FALSE) + + # prepare download options + pat <- Sys.getenv("GITHUB_PAT") + if (nzchar(Sys.which("curl")) && nzchar(pat)) { + fmt <- "--location --fail --header \"Authorization: token %s\"" + extra <- sprintf(fmt, pat) + saved <- options("download.file.method", "download.file.extra") + options(download.file.method = "curl", download.file.extra = extra) + on.exit(do.call(base::options, saved), add = TRUE) + } else if (nzchar(Sys.which("wget")) && nzchar(pat)) { + fmt <- "--header=\"Authorization: token %s\"" + extra <- sprintf(fmt, pat) + saved <- options("download.file.method", "download.file.extra") + options(download.file.method = "wget", download.file.extra = extra) + on.exit(do.call(base::options, saved), add = TRUE) + } + + message("* Downloading renv ", version, " from GitHub ... ", appendLF = FALSE) + + url <- file.path("https://api.github.com/repos/rstudio/renv/tarball", version) + name <- sprintf("renv_%s.tar.gz", version) + destfile <- file.path(tempdir(), name) + + status <- tryCatch( + renv_bootstrap_download_impl(url, destfile), + condition = identity + ) + + if (!identical(status, 0L)) { + message("FAILED") + return(FALSE) + } + + message("OK") + return(destfile) + + } + + renv_bootstrap_install <- function(version, tarball, library) { + + # attempt to install it into project library + message("* Installing renv ", version, " ... ", appendLF = FALSE) + dir.create(library, showWarnings = FALSE, recursive = TRUE) + + # invoke using system2 so we can capture and report output + bin <- R.home("bin") + exe <- if (Sys.info()[["sysname"]] == "Windows") "R.exe" else "R" + r <- file.path(bin, exe) + args <- c("--vanilla", "CMD", "INSTALL", "-l", shQuote(library), shQuote(tarball)) + output <- system2(r, args, stdout = TRUE, stderr = TRUE) + message("Done!") + + # check for successful install + status <- attr(output, "status") + if (is.numeric(status) && !identical(status, 0L)) { + header <- "Error installing renv:" + lines <- paste(rep.int("=", nchar(header)), collapse = "") + text <- c(header, lines, output) + writeLines(text, con = stderr()) + } + + status + + } + + renv_bootstrap_prefix <- function() { + + # construct version prefix + version <- paste(R.version$major, R.version$minor, sep = ".") + prefix <- paste("R", numeric_version(version)[1, 1:2], sep = "-") + + # include SVN revision for development versions of R + # (to avoid sharing platform-specific artefacts with released versions of R) + devel <- + identical(R.version[["status"]], "Under development (unstable)") || + identical(R.version[["nickname"]], "Unsuffered Consequences") + + if (devel) + prefix <- paste(prefix, R.version[["svn rev"]], sep = "-r") + + # build list of path components + components <- c(prefix, R.version$platform) + + # include prefix if provided by user + prefix <- Sys.getenv("RENV_PATHS_PREFIX") + if (nzchar(prefix)) + components <- c(prefix, components) + + # build prefix + paste(components, collapse = "/") + + } + + renv_bootstrap_library_root_name <- function(project) { + + # use project name as-is if requested + asis <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT_ASIS", unset = "FALSE") + if (asis) + return(basename(project)) + + # otherwise, disambiguate based on project's path + id <- substring(renv_bootstrap_hash_text(project), 1L, 8L) + paste(basename(project), id, sep = "-") + + } + + renv_bootstrap_library_root <- function(project) { + + path <- Sys.getenv("RENV_PATHS_LIBRARY", unset = NA) + if (!is.na(path)) + return(path) + + path <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT", unset = NA) + if (!is.na(path)) { + name <- renv_bootstrap_library_root_name(project) + return(file.path(path, name)) + } + + file.path(project, "renv/library") + + } + + renv_bootstrap_validate_version <- function(version) { + + loadedversion <- utils::packageDescription("renv", fields = "Version") + if (version == loadedversion) + return(TRUE) + + # assume four-component versions are from GitHub; three-component + # versions are from CRAN + components <- strsplit(loadedversion, "[.-]")[[1]] + remote <- if (length(components) == 4L) + paste("rstudio/renv", loadedversion, sep = "@") + else + paste("renv", loadedversion, sep = "@") + + fmt <- paste( + "renv %1$s was loaded from project library, but this project is configured to use renv %2$s.", + "Use `renv::record(\"%3$s\")` to record renv %1$s in the lockfile.", + "Use `renv::restore(packages = \"renv\")` to install renv %2$s into the project library.", + sep = "\n" + ) + + msg <- sprintf(fmt, loadedversion, version, remote) + warning(msg, call. = FALSE) + + FALSE + + } + + renv_bootstrap_hash_text <- function(text) { + + hashfile <- tempfile("renv-hash-") + on.exit(unlink(hashfile), add = TRUE) + + writeLines(text, con = hashfile) + tools::md5sum(hashfile) + + } + + renv_bootstrap_load <- function(project, libpath, version) { + + # try to load renv from the project library + if (!requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) + return(FALSE) + + # warn if the version of renv loaded does not match + renv_bootstrap_validate_version(version) + + # load the project + renv::load(project) + + TRUE + + } + + # construct path to library root + root <- renv_bootstrap_library_root(project) + + # construct library prefix for platform + prefix <- renv_bootstrap_prefix() + + # construct full libpath + libpath <- file.path(root, prefix) + + # attempt to load + if (renv_bootstrap_load(project, libpath, version)) + return(TRUE) + + # load failed; inform user we're about to bootstrap + prefix <- paste("# Bootstrapping renv", version) + postfix <- paste(rep.int("-", 77L - nchar(prefix)), collapse = "") + header <- paste(prefix, postfix) + message(header) + + # perform bootstrap + bootstrap(version, libpath) + + # exit early if we're just testing bootstrap + if (!is.na(Sys.getenv("RENV_BOOTSTRAP_INSTALL_ONLY", unset = NA))) + return(TRUE) + + # try again to load + if (requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) { + message("* Successfully installed and loaded renv ", version, ".") + return(renv::load()) + } + + # failed to download or load renv; warn the user + msg <- c( + "Failed to find an renv installation: the project will not be loaded.", + "Use `renv::activate()` to re-initialize the project." + ) + + warning(paste(msg, collapse = "\n"), call. = FALSE) + +}) diff --git a/pre_commit/resources/empty_template_environment.yml b/pre_commit/resources/empty_template_environment.yml new file mode 100644 index 00000000..0f29f0c0 --- /dev/null +++ b/pre_commit/resources/empty_template_environment.yml @@ -0,0 +1,9 @@ +channels: + - conda-forge + - defaults +dependencies: + # This cannot be empty as otherwise no environment will be created. + # We're using openssl here as it is available on all system and will + # most likely be always installed anyways. + # See https://github.com/conda/conda/issues/9487 + - openssl diff --git a/pre_commit/resources/empty_template_go.mod b/pre_commit/resources/empty_template_go.mod new file mode 100644 index 00000000..892c4e59 --- /dev/null +++ b/pre_commit/resources/empty_template_go.mod @@ -0,0 +1 @@ +module pre-commit-placeholder-empty-module diff --git a/pre_commit/resources/empty_template_package.json b/pre_commit/resources/empty_template_package.json index ac7b7259..042e9583 100644 --- a/pre_commit/resources/empty_template_package.json +++ b/pre_commit/resources/empty_template_package.json @@ -1,4 +1,4 @@ { - "name": "pre_commit_dummy_package", + "name": "pre_commit_placeholder_package", "version": "0.0.0" } diff --git a/pre_commit/resources/empty_template_pre-commit-package-dev-1.rockspec b/pre_commit/resources/empty_template_pre-commit-package-dev-1.rockspec new file mode 100644 index 00000000..f063c8e2 --- /dev/null +++ b/pre_commit/resources/empty_template_pre-commit-package-dev-1.rockspec @@ -0,0 +1,12 @@ +package = "pre-commit-package" +version = "dev-1" + +source = { + url = "git+ssh://git@github.com/pre-commit/pre-commit.git" +} +description = {} +dependencies = {} +build = { + type = "builtin", + modules = {}, +} diff --git a/pre_commit/resources/empty_template_pre_commit_dummy_package.gemspec b/pre_commit/resources/empty_template_pre_commit_dummy_package.gemspec deleted file mode 100644 index 8bfb40ca..00000000 --- a/pre_commit/resources/empty_template_pre_commit_dummy_package.gemspec +++ /dev/null @@ -1,6 +0,0 @@ -Gem::Specification.new do |s| - s.name = 'pre_commit_dummy_package' - s.version = '0.0.0' - s.summary = 'dummy gem for pre-commit hooks' - s.authors = ['Anthony Sottile'] -end diff --git a/pre_commit/resources/empty_template_pre_commit_placeholder_package.gemspec b/pre_commit/resources/empty_template_pre_commit_placeholder_package.gemspec new file mode 100644 index 00000000..630f0d4d --- /dev/null +++ b/pre_commit/resources/empty_template_pre_commit_placeholder_package.gemspec @@ -0,0 +1,6 @@ +Gem::Specification.new do |s| + s.name = 'pre_commit_placeholder_package' + s.version = '0.0.0' + s.summary = 'placeholder gem for pre-commit hooks' + s.authors = ['Anthony Sottile'] +end diff --git a/pre_commit/resources/empty_template_pubspec.yaml b/pre_commit/resources/empty_template_pubspec.yaml new file mode 100644 index 00000000..8306aeb6 --- /dev/null +++ b/pre_commit/resources/empty_template_pubspec.yaml @@ -0,0 +1,4 @@ +name: pre_commit_empty_pubspec +environment: + sdk: '>=2.12.0' +executables: {} diff --git a/pre_commit/resources/empty_template_renv.lock b/pre_commit/resources/empty_template_renv.lock new file mode 100644 index 00000000..d6e31f86 --- /dev/null +++ b/pre_commit/resources/empty_template_renv.lock @@ -0,0 +1,20 @@ +{ + "R": { + "Version": "4.0.3", + "Repositories": [ + { + "Name": "CRAN", + "URL": "https://cran.rstudio.com" + } + ] + }, + "Packages": { + "renv": { + "Package": "renv", + "Version": "0.12.5", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "5c0cdb37f063c58cdab3c7e9fbb8bd2c" + } + } +} diff --git a/pre_commit/resources/empty_template_setup.py b/pre_commit/resources/empty_template_setup.py index 68860648..e8b1ff02 100644 --- a/pre_commit/resources/empty_template_setup.py +++ b/pre_commit/resources/empty_template_setup.py @@ -1,4 +1,4 @@ from setuptools import setup -setup(name='pre-commit-dummy-package', version='0.0.0') +setup(name='pre-commit-placeholder-package', version='0.0.0', py_modules=[]) diff --git a/pre_commit/resources/hook-tmpl b/pre_commit/resources/hook-tmpl index a145c8ee..53d29f95 100755 --- a/pre_commit/resources/hook-tmpl +++ b/pre_commit/resources/hook-tmpl @@ -1,195 +1,20 @@ -#!/usr/bin/env python3 -"""File generated by pre-commit: https://pre-commit.com""" -from __future__ import print_function +#!/usr/bin/env bash +# File generated by pre-commit: https://pre-commit.com +# ID: 138fd403232d2ddd5efb44317e38bf03 -import distutils.spawn -import os -import subprocess -import sys - -# work around https://github.com/Homebrew/homebrew-core/issues/30445 -os.environ.pop('__PYVENV_LAUNCHER__', None) - -HERE = os.path.dirname(os.path.abspath(__file__)) -Z40 = '0' * 40 -ID_HASH = '138fd403232d2ddd5efb44317e38bf03' # start templated -CONFIG = None -HOOK_TYPE = None -INSTALL_PYTHON = None -SKIP_ON_MISSING_CONFIG = None +INSTALL_PYTHON='' +ARGS=(hook-impl) # end templated +HERE="$(cd "$(dirname "$0")" && pwd)" +ARGS+=(--hook-dir "$HERE" -- "$@") -class EarlyExit(RuntimeError): - pass - - -class FatalError(RuntimeError): - pass - - -def _norm_exe(exe): - """Necessary for shebang support on windows. - - roughly lifted from `identify.identify.parse_shebang` - """ - with open(exe, 'rb') as f: - if f.read(2) != b'#!': - return () - try: - first_line = f.readline().decode('UTF-8') - except UnicodeDecodeError: - return () - - cmd = first_line.split() - if cmd[0] == '/usr/bin/env': - del cmd[0] - return tuple(cmd) - - -def _run_legacy(): - if __file__.endswith('.legacy'): - raise SystemExit( - "bug: pre-commit's script is installed in migration mode\n" - 'run `pre-commit install -f --hook-type {}` to fix this\n\n' - 'Please report this bug at ' - 'https://github.com/pre-commit/pre-commit/issues'.format( - HOOK_TYPE, - ), - ) - - if HOOK_TYPE == 'pre-push': - stdin = getattr(sys.stdin, 'buffer', sys.stdin).read() - else: - stdin = None - - legacy_hook = os.path.join(HERE, '{}.legacy'.format(HOOK_TYPE)) - if os.access(legacy_hook, os.X_OK): - cmd = _norm_exe(legacy_hook) + (legacy_hook,) + tuple(sys.argv[1:]) - proc = subprocess.Popen(cmd, stdin=subprocess.PIPE if stdin else None) - proc.communicate(stdin) - return proc.returncode, stdin - else: - return 0, stdin - - -def _validate_config(): - cmd = ('git', 'rev-parse', '--show-toplevel') - top_level = subprocess.check_output(cmd).decode('UTF-8').strip() - cfg = os.path.join(top_level, CONFIG) - if os.path.isfile(cfg): - pass - elif SKIP_ON_MISSING_CONFIG or os.getenv('PRE_COMMIT_ALLOW_NO_CONFIG'): - print( - '`{}` config file not found. ' - 'Skipping `pre-commit`.'.format(CONFIG), - ) - raise EarlyExit() - else: - raise FatalError( - 'No {} file was found\n' - '- To temporarily silence this, run ' - '`PRE_COMMIT_ALLOW_NO_CONFIG=1 git ...`\n' - '- To permanently silence this, install pre-commit with the ' - '--allow-missing-config option\n' - '- To uninstall pre-commit run ' - '`pre-commit uninstall`'.format(CONFIG), - ) - - -def _exe(): - with open(os.devnull, 'wb') as devnull: - for exe in (INSTALL_PYTHON, sys.executable): - try: - if not subprocess.call( - (exe, '-c', 'import pre_commit.main'), - stdout=devnull, stderr=devnull, - ): - return (exe, '-m', 'pre_commit.main', 'run') - except OSError: - pass - - if distutils.spawn.find_executable('pre-commit'): - return ('pre-commit', 'run') - - raise FatalError( - '`pre-commit` not found. Did you forget to activate your virtualenv?', - ) - - -def _rev_exists(rev): - return not subprocess.call(('git', 'rev-list', '--quiet', rev)) - - -def _pre_push(stdin): - remote = sys.argv[1] - - opts = () - for line in stdin.decode('UTF-8').splitlines(): - _, local_sha, _, remote_sha = line.split() - if local_sha == Z40: - continue - elif remote_sha != Z40 and _rev_exists(remote_sha): - opts = ('--origin', local_sha, '--source', remote_sha) - else: - # ancestors not found in remote - ancestors = subprocess.check_output(( - 'git', 'rev-list', local_sha, '--topo-order', '--reverse', - '--not', '--remotes={}'.format(remote), - )).decode().strip() - if not ancestors: - continue - else: - first_ancestor = ancestors.splitlines()[0] - cmd = ('git', 'rev-list', '--max-parents=0', local_sha) - roots = set(subprocess.check_output(cmd).decode().splitlines()) - if first_ancestor in roots: - # pushing the whole tree including root commit - opts = ('--all-files',) - else: - cmd = ('git', 'rev-parse', '{}^'.format(first_ancestor)) - source = subprocess.check_output(cmd).decode().strip() - opts = ('--origin', local_sha, '--source', source) - - if opts: - return opts - else: - # An attempt to push an empty changeset - raise EarlyExit() - - -def _opts(stdin): - fns = { - 'prepare-commit-msg': lambda _: ('--commit-msg-filename', sys.argv[1]), - 'commit-msg': lambda _: ('--commit-msg-filename', sys.argv[1]), - 'pre-commit': lambda _: (), - 'pre-push': _pre_push, - } - stage = HOOK_TYPE.replace('pre-', '') - return ('--config', CONFIG, '--hook-stage', stage) + fns[HOOK_TYPE](stdin) - - -if sys.version_info < (3, 7): # https://bugs.python.org/issue25942 - def _subprocess_call(cmd): # this is the python 2.7 implementation - return subprocess.Popen(cmd).wait() -else: - _subprocess_call = subprocess.call - - -def main(): - retv, stdin = _run_legacy() - try: - _validate_config() - return retv | _subprocess_call(_exe() + _opts(stdin)) - except EarlyExit: - return retv - except FatalError as e: - print(e.args[0]) - return 1 - except KeyboardInterrupt: - return 1 - - -if __name__ == '__main__': - exit(main()) +if [ -x "$INSTALL_PYTHON" ]; then + exec "$INSTALL_PYTHON" -mpre_commit "${ARGS[@]}" +elif command -v pre-commit > /dev/null; then + exec pre-commit "${ARGS[@]}" +else + echo '`pre-commit` not found. Did you forget to activate your virtualenv?' 1>&2 + exit 1 +fi diff --git a/pre_commit/resources/rbenv.tar.gz b/pre_commit/resources/rbenv.tar.gz index 4505e471..b5df0874 100644 Binary files a/pre_commit/resources/rbenv.tar.gz and b/pre_commit/resources/rbenv.tar.gz differ diff --git a/pre_commit/resources/ruby-build.tar.gz b/pre_commit/resources/ruby-build.tar.gz index 66107ddf..5c82c906 100644 Binary files a/pre_commit/resources/ruby-build.tar.gz and b/pre_commit/resources/ruby-build.tar.gz differ diff --git a/pre_commit/resources/ruby-download.tar.gz b/pre_commit/resources/ruby-download.tar.gz index 7ccfb6c8..f7cb0b42 100644 Binary files a/pre_commit/resources/ruby-download.tar.gz and b/pre_commit/resources/ruby-download.tar.gz differ diff --git a/pre_commit/staged_files_only.py b/pre_commit/staged_files_only.py index 7af319d7..99ea0979 100644 --- a/pre_commit/staged_files_only.py +++ b/pre_commit/staged_files_only.py @@ -1,32 +1,39 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib -import io import logging import os.path import time +from collections.abc import Generator from pre_commit import git +from pre_commit.errors import FatalError from pre_commit.util import CalledProcessError from pre_commit.util import cmd_output -from pre_commit.util import mkdirp +from pre_commit.util import cmd_output_b from pre_commit.xargs import xargs logger = logging.getLogger('pre_commit') +# without forcing submodule.recurse=0, changes in nested submodules will be +# discarded if `submodule.recurse=1` is configured +# we choose this instead of `--no-recurse-submodules` because it works on +# versions of git before that option was added to `git checkout` +_CHECKOUT_CMD = ('git', '-c', 'submodule.recurse=0', 'checkout', '--', '.') -def _git_apply(patch): + +def _git_apply(patch: str) -> None: args = ('apply', '--whitespace=nowarn', patch) try: - cmd_output('git', *args, encoding=None) + cmd_output_b('git', *args) except CalledProcessError: # Retry with autocrlf=false -- see #570 - cmd_output('git', '-c', 'core.autocrlf=false', *args, encoding=None) + cmd_output_b('git', '-c', 'core.autocrlf=false', *args) @contextlib.contextmanager -def _intent_to_add_cleared(): +def _intent_to_add_cleared() -> Generator[None]: intent_to_add = git.intent_to_add_files() if intent_to_add: logger.warning('Unstaged intent-to-add files detected.') @@ -41,29 +48,37 @@ def _intent_to_add_cleared(): @contextlib.contextmanager -def _unstaged_changes_cleared(patch_dir): +def _unstaged_changes_cleared(patch_dir: str) -> Generator[None]: tree = cmd_output('git', 'write-tree')[1].strip() - retcode, diff_stdout_binary, _ = cmd_output( + diff_cmd = ( 'git', 'diff-index', '--ignore-submodules', '--binary', '--exit-code', '--no-color', '--no-ext-diff', tree, '--', - retcode=None, - encoding=None, ) - if retcode and diff_stdout_binary.strip(): - patch_filename = 'patch{}'.format(int(time.time())) + retcode, diff_stdout, diff_stderr = cmd_output_b(*diff_cmd, check=False) + if retcode == 0: + # There weren't any staged files so we don't need to do anything + # special + yield + elif retcode == 1 and not diff_stdout.strip(): + # due to behaviour (probably a bug?) in git with crlf endings and + # autocrlf set to either `true` or `input` sometimes git will refuse + # to show a crlf-only diff to us :( + yield + elif retcode == 1 and diff_stdout.strip(): + patch_filename = f'patch{int(time.time())}-{os.getpid()}' patch_filename = os.path.join(patch_dir, patch_filename) logger.warning('Unstaged files detected.') - logger.info( - 'Stashing unstaged files to {}.'.format(patch_filename), - ) + logger.info(f'Stashing unstaged files to {patch_filename}.') # Save the current unstaged changes as a patch - mkdirp(patch_dir) - with io.open(patch_filename, 'wb') as patch_file: - patch_file.write(diff_stdout_binary) + os.makedirs(patch_dir, exist_ok=True) + with open(patch_filename, 'wb') as patch_file: + patch_file.write(diff_stdout) + + # prevent recursive post-checkout hooks (#1418) + no_checkout_env = dict(os.environ, _PRE_COMMIT_SKIP_POST_CHECKOUT='1') - # Clear the working directory of unstaged changes - cmd_output('git', 'checkout', '--', '.') try: + cmd_output_b(*_CHECKOUT_CMD, env=no_checkout_env) yield finally: # Try to apply the patch we saved @@ -77,17 +92,20 @@ def _unstaged_changes_cleared(patch_dir): # We failed to apply the patch, presumably due to fixes made # by hooks. # Roll back the changes made by hooks. - cmd_output('git', 'checkout', '--', '.') + cmd_output_b(*_CHECKOUT_CMD, env=no_checkout_env) _git_apply(patch_filename) - logger.info('Restored changes from {}.'.format(patch_filename)) - else: - # There weren't any staged files so we don't need to do anything - # special - yield + + logger.info(f'Restored changes from {patch_filename}.') + else: # pragma: win32 no cover + # some error occurred while requesting the diff + e = CalledProcessError(retcode, diff_cmd, b'', diff_stderr) + raise FatalError( + f'pre-commit failed to diff -- perhaps due to permissions?\n\n{e}', + ) @contextlib.contextmanager -def staged_files_only(patch_dir): +def staged_files_only(patch_dir: str) -> Generator[None]: """Clear any unstaged changes from the git working directory inside this context. """ diff --git a/pre_commit/store.py b/pre_commit/store.py index 55c57a3e..dc90c051 100644 --- a/pre_commit/store.py +++ b/pre_commit/store.py @@ -1,48 +1,75 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib -import io import logging import os.path import sqlite3 import tempfile +from collections.abc import Callable +from collections.abc import Generator +from collections.abc import Sequence import pre_commit.constants as C +from pre_commit import clientlib from pre_commit import file_lock from pre_commit import git from pre_commit.util import CalledProcessError from pre_commit.util import clean_path_on_failure -from pre_commit.util import cmd_output +from pre_commit.util import cmd_output_b from pre_commit.util import resource_text -from pre_commit.util import rmtree logger = logging.getLogger('pre_commit') -def _get_default_directory(): +def _get_default_directory() -> str: """Returns the default directory for the Store. This is intentionally underscored to indicate that `Store.get_default_directory` is the intended way to get this information. This is also done so `Store.get_default_directory` can be mocked in tests and `_get_default_directory` can be tested. """ - return os.environ.get('PRE_COMMIT_HOME') or os.path.join( + ret = os.environ.get('PRE_COMMIT_HOME') or os.path.join( os.environ.get('XDG_CACHE_HOME') or os.path.expanduser('~/.cache'), 'pre-commit', ) + return os.path.realpath(ret) -class Store(object): +_LOCAL_RESOURCES = ( + 'Cargo.toml', 'main.go', 'go.mod', 'main.rs', '.npmignore', + 'package.json', 'pre-commit-package-dev-1.rockspec', + 'pre_commit_placeholder_package.gemspec', 'setup.py', + 'environment.yml', 'Makefile.PL', 'pubspec.yaml', + 'renv.lock', 'renv/activate.R', 'renv/LICENSE.renv', +) + + +def _make_local_repo(directory: str) -> None: + for resource in _LOCAL_RESOURCES: + resource_dirname, resource_basename = os.path.split(resource) + contents = resource_text(f'empty_template_{resource_basename}') + target_dir = os.path.join(directory, resource_dirname) + target_file = os.path.join(target_dir, resource_basename) + os.makedirs(target_dir, exist_ok=True) + with open(target_file, 'w') as f: + f.write(contents) + + +class Store: get_default_directory = staticmethod(_get_default_directory) - def __init__(self, directory=None): + def __init__(self, directory: str | None = None) -> None: self.directory = directory or Store.get_default_directory() self.db_path = os.path.join(self.directory, 'db.db') + self.readonly = ( + os.path.exists(self.directory) and + not os.access(self.directory, os.W_OK) + ) if not os.path.exists(self.directory): - os.makedirs(self.directory) - with io.open(os.path.join(self.directory, 'README'), 'w') as f: + os.makedirs(self.directory, exist_ok=True) + with open(os.path.join(self.directory, 'README'), 'w') as f: f.write( 'This directory is maintained by the pre-commit project.\n' 'Learn more: https://github.com/pre-commit/pre-commit\n', @@ -68,21 +95,24 @@ class Store(object): ' PRIMARY KEY (repo, ref)' ');', ) - self._create_config_table_if_not_exists(db) + self._create_configs_table(db) # Atomic file move - os.rename(tmpfile, self.db_path) + os.replace(tmpfile, self.db_path) @contextlib.contextmanager - def exclusive_lock(self): - def blocked_cb(): # pragma: no cover (tests are single-process) + def exclusive_lock(self) -> Generator[None]: + def blocked_cb() -> None: # pragma: no cover (tests are in-process) logger.info('Locking pre-commit directory') with file_lock.lock(os.path.join(self.directory, '.lock'), blocked_cb): yield @contextlib.contextmanager - def connect(self, db_path=None): + def connect( + self, + db_path: str | None = None, + ) -> Generator[sqlite3.Connection]: db_path = db_path or self.db_path # sqlite doesn't close its fd with its contextmanager >.< # contextlib.closing fixes this. @@ -93,24 +123,30 @@ class Store(object): yield db @classmethod - def db_repo_name(cls, repo, deps): + def db_repo_name(cls, repo: str, deps: Sequence[str]) -> str: if deps: - return '{}:{}'.format(repo, ','.join(sorted(deps))) + return f'{repo}:{",".join(deps)}' else: return repo - def _new_repo(self, repo, ref, deps, make_strategy): + def _new_repo( + self, + repo: str, + ref: str, + deps: Sequence[str], + make_strategy: Callable[[str], None], + ) -> str: + original_repo = repo repo = self.db_repo_name(repo, deps) - def _get_result(): + def _get_result() -> str | None: # Check if we already exist with self.connect() as db: result = db.execute( 'SELECT path FROM repos WHERE repo = ? AND ref = ?', (repo, ref), ).fetchone() - if result: - return result[0] + return result[0] if result else None result = _get_result() if result: @@ -121,7 +157,7 @@ class Store(object): if result: # pragma: no cover (race) return result - logger.info('Initializing environment for {}.'.format(repo)) + logger.info(f'Initializing environment for {repo}.') directory = tempfile.mkdtemp(prefix='repo', dir=self.directory) with clean_path_on_failure(directory): @@ -133,16 +169,19 @@ class Store(object): 'INSERT INTO repos (repo, ref, path) VALUES (?, ?, ?)', [repo, ref, directory], ) + + clientlib.warn_for_stages_on_repo_init(original_repo, directory) + return directory - def _complete_clone(self, ref, git_cmd): + def _complete_clone(self, ref: str, git_cmd: Callable[..., None]) -> None: """Perform a complete clone of a repository and its submodules """ git_cmd('fetch', 'origin', '--tags') git_cmd('checkout', ref) git_cmd('submodule', 'update', '--init', '--recursive') - def _shallow_clone(self, ref, git_cmd): + def _shallow_clone(self, ref: str, git_cmd: Callable[..., None]) -> None: """Perform a shallow clone of a repository and its submodules """ git_config = 'protocol.version=2' @@ -153,15 +192,15 @@ class Store(object): '--depth=1', ) - def clone(self, repo, ref, deps=()): + def clone(self, repo: str, ref: str, deps: Sequence[str] = ()) -> str: """Clone the given url and checkout the specific ref.""" - def clone_strategy(directory): + def clone_strategy(directory: str) -> None: git.init_repo(directory, repo) env = git.no_git_env() - def _git_cmd(*args): - cmd_output('git', *args, cwd=directory, env=env) + def _git_cmd(*args: str) -> None: + cmd_output_b('git', *args, cwd=directory, env=env) try: self._shallow_clone(ref, _git_cmd) @@ -170,33 +209,12 @@ class Store(object): return self._new_repo(repo, ref, deps, clone_strategy) - LOCAL_RESOURCES = ( - 'Cargo.toml', 'main.go', 'main.rs', '.npmignore', 'package.json', - 'pre_commit_dummy_package.gemspec', 'setup.py', - ) - - def make_local(self, deps): - def make_local_strategy(directory): - for resource in self.LOCAL_RESOURCES: - contents = resource_text('empty_template_{}'.format(resource)) - with io.open(os.path.join(directory, resource), 'w') as f: - f.write(contents) - - env = git.no_git_env() - - # initialize the git repository so it looks more like cloned repos - def _git_cmd(*args): - cmd_output('git', *args, cwd=directory, env=env) - - git.init_repo(directory, '<>') - _git_cmd('add', '.') - git.commit(repo=directory) - + def make_local(self, deps: Sequence[str]) -> str: return self._new_repo( - 'local', C.LOCAL_REPO_VERSION, deps, make_local_strategy, + 'local', C.LOCAL_REPO_VERSION, deps, _make_local_repo, ) - def _create_config_table_if_not_exists(self, db): + def _create_configs_table(self, db: sqlite3.Connection) -> None: db.executescript( 'CREATE TABLE IF NOT EXISTS configs (' ' path TEXT NOT NULL,' @@ -204,35 +222,14 @@ class Store(object): ');', ) - def mark_config_used(self, path): + def mark_config_used(self, path: str) -> None: + if self.readonly: # pragma: win32 no cover + return path = os.path.realpath(path) # don't insert config files that do not exist if not os.path.exists(path): return with self.connect() as db: # TODO: eventually remove this and only create in _create - self._create_config_table_if_not_exists(db) + self._create_configs_table(db) db.execute('INSERT OR IGNORE INTO configs VALUES (?)', (path,)) - - def select_all_configs(self): - with self.connect() as db: - self._create_config_table_if_not_exists(db) - rows = db.execute('SELECT path FROM configs').fetchall() - return [path for path, in rows] - - def delete_configs(self, configs): - with self.connect() as db: - rows = [(path,) for path in configs] - db.executemany('DELETE FROM configs WHERE path = ?', rows) - - def select_all_repos(self): - with self.connect() as db: - return db.execute('SELECT repo, ref, path from repos').fetchall() - - def delete_repo(self, db_repo_name, ref, path): - with self.connect() as db: - db.execute( - 'DELETE FROM repos WHERE repo = ? and ref = ?', - (db_repo_name, ref), - ) - rmtree(path) diff --git a/pre_commit/util.py b/pre_commit/util.py index eb5411fd..19b1880b 100644 --- a/pre_commit/util.py +++ b/pre_commit/util.py @@ -1,37 +1,31 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib import errno +import importlib.resources import os.path import shutil import stat import subprocess import sys -import tempfile +from collections.abc import Callable +from collections.abc import Generator +from types import TracebackType +from typing import Any -import six - -from pre_commit import five from pre_commit import parse_shebang -if sys.version_info >= (3, 7): # pragma: no cover (PY37+) - from importlib.resources import open_binary - from importlib.resources import read_text -else: # pragma: no cover ( bytes: + with contextlib.suppress(TypeError): + return bytes(exc) + with contextlib.suppress(Exception): + return str(exc).encode() + return f''.encode() @contextlib.contextmanager -def clean_path_on_failure(path): +def clean_path_on_failure(path: str) -> Generator[None]: """Cleans up the directory on an exceptional failure.""" try: yield @@ -41,137 +35,205 @@ def clean_path_on_failure(path): raise -@contextlib.contextmanager -def noop_context(): - yield +def resource_text(filename: str) -> str: + files = importlib.resources.files('pre_commit.resources') + return files.joinpath(filename).read_text() -@contextlib.contextmanager -def tmpdir(): - """Contextmanager to create a temporary directory. It will be cleaned up - afterwards. - """ - tempdir = tempfile.mkdtemp() - try: - yield tempdir - finally: - rmtree(tempdir) - - -def resource_bytesio(filename): - return open_binary('pre_commit.resources', filename) - - -def resource_text(filename): - return read_text('pre_commit.resources', filename) - - -def make_executable(filename): +def make_executable(filename: str) -> None: original_mode = os.stat(filename).st_mode - os.chmod( - filename, original_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH, - ) + new_mode = original_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH + os.chmod(filename, new_mode) class CalledProcessError(RuntimeError): - def __init__(self, returncode, cmd, expected_returncode, output=None): - super(CalledProcessError, self).__init__( - returncode, cmd, expected_returncode, output, - ) + def __init__( + self, + returncode: int, + cmd: tuple[str, ...], + stdout: bytes, + stderr: bytes | None, + ) -> None: + super().__init__(returncode, cmd, stdout, stderr) self.returncode = returncode self.cmd = cmd - self.expected_returncode = expected_returncode - self.output = output + self.stdout = stdout + self.stderr = stderr - def to_bytes(self): - output = [] - for maybe_text in self.output: - if maybe_text: - output.append( - b'\n ' + - five.to_bytes(maybe_text).replace(b'\n', b'\n '), - ) + def __bytes__(self) -> bytes: + def _indent_or_none(part: bytes | None) -> bytes: + if part: + return b'\n ' + part.replace(b'\n', b'\n ').rstrip() else: - output.append(b'(none)') + return b' (none)' return b''.join(( - five.to_bytes( - 'Command: {!r}\n' - 'Return code: {}\n' - 'Expected return code: {}\n'.format( - self.cmd, self.returncode, self.expected_returncode, - ), - ), - b'Output: ', output[0], b'\n', - b'Errors: ', output[1], b'\n', + f'command: {self.cmd!r}\n'.encode(), + f'return code: {self.returncode}\n'.encode(), + b'stdout:', _indent_or_none(self.stdout), b'\n', + b'stderr:', _indent_or_none(self.stderr), )) - def to_text(self): - return self.to_bytes().decode('UTF-8') - - if six.PY2: # pragma: no cover (py2) - __str__ = to_bytes - __unicode__ = to_text - else: # pragma: no cover (py3) - __bytes__ = to_bytes - __str__ = to_text + def __str__(self) -> str: + return self.__bytes__().decode() -def cmd_output(*cmd, **kwargs): - retcode = kwargs.pop('retcode', 0) - encoding = kwargs.pop('encoding', 'UTF-8') +def _setdefault_kwargs(kwargs: dict[str, Any]) -> None: + for arg in ('stdin', 'stdout', 'stderr'): + kwargs.setdefault(arg, subprocess.PIPE) - popen_kwargs = { - 'stdin': subprocess.PIPE, - 'stdout': subprocess.PIPE, - 'stderr': subprocess.PIPE, - } - # py2/py3 on windows are more strict about the types here - cmd = tuple(five.n(arg) for arg in cmd) - kwargs['env'] = { - five.n(key): five.n(value) - for key, value in kwargs.pop('env', {}).items() - } or None +def _oserror_to_output(e: OSError) -> tuple[int, bytes, None]: + return 1, force_bytes(e).rstrip(b'\n') + b'\n', None + + +def cmd_output_b( + *cmd: str, + check: bool = True, + **kwargs: Any, +) -> tuple[int, bytes, bytes | None]: + _setdefault_kwargs(kwargs) try: - cmd = parse_shebang.normalize_cmd(cmd) + cmd = parse_shebang.normalize_cmd(cmd, env=kwargs.get('env')) except parse_shebang.ExecutableNotFoundError as e: - returncode, stdout, stderr = e.to_output() + returncode, stdout_b, stderr_b = e.to_output() else: - popen_kwargs.update(kwargs) - proc = subprocess.Popen(cmd, **popen_kwargs) - stdout, stderr = proc.communicate() - returncode = proc.returncode - if encoding is not None and stdout is not None: - stdout = stdout.decode(encoding) - if encoding is not None and stderr is not None: - stderr = stderr.decode(encoding) + try: + proc = subprocess.Popen(cmd, **kwargs) + except OSError as e: + returncode, stdout_b, stderr_b = _oserror_to_output(e) + else: + stdout_b, stderr_b = proc.communicate() + returncode = proc.returncode - if retcode is not None and retcode != returncode: - raise CalledProcessError( - returncode, cmd, retcode, output=(stdout, stderr), - ) + if check and returncode: + raise CalledProcessError(returncode, cmd, stdout_b, stderr_b) + return returncode, stdout_b, stderr_b + + +def cmd_output(*cmd: str, **kwargs: Any) -> tuple[int, str, str | None]: + returncode, stdout_b, stderr_b = cmd_output_b(*cmd, **kwargs) + stdout = stdout_b.decode() if stdout_b is not None else None + stderr = stderr_b.decode() if stderr_b is not None else None return returncode, stdout, stderr -def rmtree(path): - """On windows, rmtree fails for readonly dirs.""" - def handle_remove_readonly(func, path, exc): - excvalue = exc[1] - if ( - func in (os.rmdir, os.remove, os.unlink) and - excvalue.errno == errno.EACCES - ): - for p in (path, os.path.dirname(path)): - os.chmod(p, os.stat(p).st_mode | stat.S_IWUSR) - func(path) - else: - raise - shutil.rmtree(path, ignore_errors=False, onerror=handle_remove_readonly) +if sys.platform != 'win32': # pragma: win32 no cover + from os import openpty + import termios + + class Pty: + def __init__(self) -> None: + self.r: int | None = None + self.w: int | None = None + + def __enter__(self) -> Pty: + self.r, self.w = openpty() + + # tty flags normally change \n to \r\n + attrs = termios.tcgetattr(self.w) + assert isinstance(attrs[1], int) + attrs[1] &= ~(termios.ONLCR | termios.OPOST) + termios.tcsetattr(self.w, termios.TCSANOW, attrs) + + return self + + def close_w(self) -> None: + if self.w is not None: + os.close(self.w) + self.w = None + + def close_r(self) -> None: + assert self.r is not None + os.close(self.r) + self.r = None + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None: + self.close_w() + self.close_r() + + def cmd_output_p( + *cmd: str, + check: bool = True, + **kwargs: Any, + ) -> tuple[int, bytes, bytes | None]: + assert check is False + assert kwargs['stderr'] == subprocess.STDOUT, kwargs['stderr'] + _setdefault_kwargs(kwargs) + + try: + cmd = parse_shebang.normalize_cmd(cmd) + except parse_shebang.ExecutableNotFoundError as e: + return e.to_output() + + with open(os.devnull) as devnull, Pty() as pty: + assert pty.r is not None + kwargs.update({'stdin': devnull, 'stdout': pty.w, 'stderr': pty.w}) + try: + proc = subprocess.Popen(cmd, **kwargs) + except OSError as e: + return _oserror_to_output(e) + + pty.close_w() + + buf = b'' + while True: + try: + bts = os.read(pty.r, 4096) + except OSError as e: + if e.errno == errno.EIO: + bts = b'' + else: + raise + else: + buf += bts + if not bts: + break + + return proc.wait(), buf, None +else: # pragma: no cover + cmd_output_p = cmd_output_b -def parse_version(s): - """poor man's version comparison""" - return tuple(int(p) for p in s.split('.')) +def _handle_readonly( + func: Callable[[str], object], + path: str, + exc: BaseException, +) -> None: + if ( + func in (os.rmdir, os.remove, os.unlink) and + isinstance(exc, OSError) and + exc.errno in {errno.EACCES, errno.EPERM} + ): + for p in (path, os.path.dirname(path)): + os.chmod(p, os.stat(p).st_mode | stat.S_IWUSR) + func(path) + else: + raise + + +if sys.version_info < (3, 12): # pragma: <3.12 cover + def _handle_readonly_old( + func: Callable[[str], object], + path: str, + excinfo: tuple[type[BaseException], BaseException, TracebackType], + ) -> None: + return _handle_readonly(func, path, excinfo[1]) + + def rmtree(path: str) -> None: + shutil.rmtree(path, ignore_errors=False, onerror=_handle_readonly_old) +else: # pragma: >=3.12 cover + def rmtree(path: str) -> None: + """On windows, rmtree fails for readonly dirs.""" + shutil.rmtree(path, ignore_errors=False, onexc=_handle_readonly) + + +def win_exe(s: str) -> str: + return s if sys.platform != 'win32' else f'{s}.exe' diff --git a/pre_commit/xargs.py b/pre_commit/xargs.py index 936a5bef..7c98d167 100644 --- a/pre_commit/xargs.py +++ b/pre_commit/xargs.py @@ -1,20 +1,44 @@ -from __future__ import absolute_import -from __future__ import division -from __future__ import unicode_literals +from __future__ import annotations import concurrent.futures import contextlib import math +import multiprocessing import os +import subprocess import sys - -import six +from collections.abc import Callable +from collections.abc import Generator +from collections.abc import Iterable +from collections.abc import MutableMapping +from collections.abc import Sequence +from typing import Any +from typing import TypeVar from pre_commit import parse_shebang -from pre_commit.util import cmd_output +from pre_commit.util import cmd_output_b +from pre_commit.util import cmd_output_p + +TArg = TypeVar('TArg') +TRet = TypeVar('TRet') -def _environ_size(_env=None): +def cpu_count() -> int: + try: + # On systems that support it, this will return a more accurate count of + # usable CPUs for the current process, which will take into account + # cgroup limits + return len(os.sched_getaffinity(0)) + except AttributeError: + pass + + try: + return multiprocessing.cpu_count() + except NotImplementedError: + return 1 + + +def _environ_size(_env: MutableMapping[str, str] | None = None) -> int: environ = _env if _env is not None else getattr(os, 'environb', os.environ) size = 8 * len(environ) # number of pointers in `envp` for k, v in environ.items(): @@ -22,9 +46,9 @@ def _environ_size(_env=None): return size -def _get_platform_max_length(): # pragma: no cover (platform specific) +def _get_platform_max_length() -> int: # pragma: no cover (platform specific) if os.name == 'posix': - maximum = os.sysconf(str('SC_ARG_MAX')) - 2048 - _environ_size() + maximum = os.sysconf('SC_ARG_MAX') - 2048 - _environ_size() maximum = max(min(maximum, 2 ** 17), 2 ** 12) return maximum elif os.name == 'nt': @@ -34,17 +58,13 @@ def _get_platform_max_length(): # pragma: no cover (platform specific) return 2 ** 12 -def _command_length(*cmd): +def _command_length(*cmd: str) -> int: full_cmd = ' '.join(cmd) # win32 uses the amount of characters, more details at: # https://github.com/pre-commit/pre-commit/pull/839 if sys.platform == 'win32': - # the python2.x apis require bytes, we encode as UTF-8 - if six.PY2: - return len(full_cmd.encode('utf-8')) - else: - return len(full_cmd.encode('utf-16le')) // 2 + return len(full_cmd.encode('utf-16le')) // 2 else: return len(full_cmd.encode(sys.getfilesystemencoding())) @@ -53,7 +73,12 @@ class ArgumentTooLongError(RuntimeError): pass -def partition(cmd, varargs, target_concurrency, _max_length=None): +def partition( + cmd: Sequence[str], + varargs: Sequence[str], + target_concurrency: int, + _max_length: int | None = None, +) -> tuple[tuple[str, ...], ...]: _max_length = _max_length or _get_platform_max_length() # Generally, we try to partition evenly into at least `target_concurrency` @@ -63,7 +88,7 @@ def partition(cmd, varargs, target_concurrency, _max_length=None): cmd = tuple(cmd) ret = [] - ret_cmd = [] + ret_cmd: list[str] = [] # Reversed so arguments are in order varargs = list(reversed(varargs)) @@ -93,7 +118,9 @@ def partition(cmd, varargs, target_concurrency, _max_length=None): @contextlib.contextmanager -def _thread_mapper(maxsize): +def _thread_mapper(maxsize: int) -> Generator[ + Callable[[Callable[[TArg], TRet], Iterable[TArg]], Iterable[TRet]], +]: if maxsize == 1: yield map else: @@ -101,46 +128,57 @@ def _thread_mapper(maxsize): yield ex.map -def xargs(cmd, varargs, **kwargs): +def xargs( + cmd: tuple[str, ...], + varargs: Sequence[str], + *, + color: bool = False, + target_concurrency: int = 1, + _max_length: int = _get_platform_max_length(), + **kwargs: Any, +) -> tuple[int, bytes]: """A simplified implementation of xargs. - negate: Make nonzero successful and zero a failure + color: Make a pty if on a platform that supports it target_concurrency: Target number of partitions to run concurrently """ - negate = kwargs.pop('negate', False) - target_concurrency = kwargs.pop('target_concurrency', 1) - max_length = kwargs.pop('_max_length', _get_platform_max_length()) + cmd_fn = cmd_output_p if color else cmd_output_b retcode = 0 stdout = b'' - stderr = b'' try: cmd = parse_shebang.normalize_cmd(cmd) except parse_shebang.ExecutableNotFoundError as e: - return e.to_output() + return e.to_output()[:2] - partitions = partition(cmd, varargs, target_concurrency, max_length) + # on windows, batch files have a separate length limit than windows itself + if ( + sys.platform == 'win32' and + cmd[0].lower().endswith(('.bat', '.cmd')) + ): # pragma: win32 cover + # this is implementation details but the command gets translated into + # full/path/to/cmd.exe /c *cmd + cmd_exe = parse_shebang.find_executable('cmd.exe') + # 1024 is additionally subtracted to give headroom for further + # expansion inside the batch file + _max_length = 8192 - len(cmd_exe) - len(' /c ') - 1024 - def run_cmd_partition(run_cmd): - return cmd_output(*run_cmd, encoding=None, retcode=None, **kwargs) + partitions = partition(cmd, varargs, target_concurrency, _max_length) + + def run_cmd_partition( + run_cmd: tuple[str, ...], + ) -> tuple[int, bytes, bytes | None]: + return cmd_fn( + *run_cmd, check=False, stderr=subprocess.STDOUT, **kwargs, + ) threads = min(len(partitions), target_concurrency) with _thread_mapper(threads) as thread_map: results = thread_map(run_cmd_partition, partitions) - for proc_retcode, proc_out, proc_err in results: - # This is *slightly* too clever so I'll explain it. - # First the xor boolean table: - # T | F | - # +-------+ - # T | F | T | - # --+-------+ - # F | T | F | - # --+-------+ - # When negate is True, it has the effect of flipping the return - # code. Otherwise, the returncode is unchanged. - retcode |= bool(proc_retcode) ^ negate + for proc_retcode, proc_out, _ in results: + if abs(proc_retcode) > abs(retcode): + retcode = proc_retcode stdout += proc_out - stderr += proc_err - return retcode, stdout, stderr + return retcode, stdout diff --git a/pre_commit/yaml.py b/pre_commit/yaml.py new file mode 100644 index 00000000..a5bbbc99 --- /dev/null +++ b/pre_commit/yaml.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +import functools +from typing import Any + +import yaml + +Loader = getattr(yaml, 'CSafeLoader', yaml.SafeLoader) +yaml_compose = functools.partial(yaml.compose, Loader=Loader) +yaml_load = functools.partial(yaml.load, Loader=Loader) +Dumper = getattr(yaml, 'CSafeDumper', yaml.SafeDumper) + + +def yaml_dump(o: Any, **kwargs: Any) -> str: + # when python/mypy#1484 is solved, this can be `functools.partial` + return yaml.dump( + o, Dumper=Dumper, default_flow_style=False, indent=4, sort_keys=False, + **kwargs, + ) diff --git a/pre_commit/yaml_rewrite.py b/pre_commit/yaml_rewrite.py new file mode 100644 index 00000000..8d0e8fdb --- /dev/null +++ b/pre_commit/yaml_rewrite.py @@ -0,0 +1,52 @@ +from __future__ import annotations + +from collections.abc import Generator +from collections.abc import Iterable +from typing import NamedTuple +from typing import Protocol + +from yaml.nodes import MappingNode +from yaml.nodes import Node +from yaml.nodes import ScalarNode +from yaml.nodes import SequenceNode + + +class _Matcher(Protocol): + def match(self, n: Node) -> Generator[Node]: ... + + +class MappingKey(NamedTuple): + k: str + + def match(self, n: Node) -> Generator[Node]: + if isinstance(n, MappingNode): + for k, _ in n.value: + if k.value == self.k: + yield k + + +class MappingValue(NamedTuple): + k: str + + def match(self, n: Node) -> Generator[Node]: + if isinstance(n, MappingNode): + for k, v in n.value: + if k.value == self.k: + yield v + + +class SequenceItem(NamedTuple): + def match(self, n: Node) -> Generator[Node]: + if isinstance(n, SequenceNode): + yield from n.value + + +def _match(gen: Iterable[Node], m: _Matcher) -> Iterable[Node]: + return (n for src in gen for n in m.match(src)) + + +def match(n: Node, matcher: tuple[_Matcher, ...]) -> Generator[ScalarNode]: + gen: Iterable[Node] = (n,) + for m in matcher: + gen = _match(gen, m) + return (n for n in gen if isinstance(n, ScalarNode)) diff --git a/requirements-dev.txt b/requirements-dev.txt index 157f287d..a23a3730 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,7 +1,6 @@ --e . - +covdefaults>=2.2 coverage -flake8 -mock +distlib pytest pytest-env +re-assert diff --git a/setup.cfg b/setup.cfg index c7175b24..a95ee447 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = pre_commit -version = 1.18.1 +version = 4.5.1 description = A framework for managing and maintaining multi-language pre-commit hooks. long_description = file: README.md long_description_content_type = text/markdown @@ -8,40 +8,31 @@ url = https://github.com/pre-commit/pre-commit author = Anthony Sottile author_email = asottile@umich.edu license = MIT -license_file = LICENSE +license_files = LICENSE classifiers = - License :: OSI Approved :: MIT License - Programming Language :: Python :: 2 - Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 - Programming Language :: Python :: 3.4 - Programming Language :: Python :: 3.5 - Programming Language :: Python :: 3.6 - Programming Language :: Python :: 3.7 + Programming Language :: Python :: 3 :: Only Programming Language :: Python :: Implementation :: CPython Programming Language :: Python :: Implementation :: PyPy [options] packages = find: install_requires = - aspy.yaml cfgv>=2.0.0 identify>=1.0.0 - importlib-metadata nodeenv>=0.11.1 - pyyaml - six - toml - virtualenv>=15.2 - futures; python_version<"3.2" - importlib-resources; python_version<"3.7" -python_requires = >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.* + pyyaml>=5.1 + virtualenv>=20.10.0 +python_requires = >=3.10 + +[options.packages.find] +exclude = + tests* + testing* [options.entry_points] console_scripts = pre-commit = pre_commit.main:main - pre-commit-validate-config = pre_commit.clientlib:validate_config_main - pre-commit-validate-manifest = pre_commit.clientlib:validate_manifest_main [options.package_data] pre_commit.resources = @@ -49,10 +40,24 @@ pre_commit.resources = empty_template_* hook-tmpl -[options.packages.find] -exclude = - tests* - testing* - [bdist_wheel] universal = True + +[coverage:run] +plugins = covdefaults +omit = pre_commit/resources/* + +[mypy] +check_untyped_defs = true +disallow_any_generics = true +disallow_incomplete_defs = true +disallow_untyped_defs = true +enable_error_code = deprecated +warn_redundant_casts = true +warn_unused_ignores = true + +[mypy-testing.*] +disallow_untyped_defs = false + +[mypy-tests.*] +disallow_untyped_defs = false diff --git a/setup.py b/setup.py index 8bf1ba93..3d93aefb 100644 --- a/setup.py +++ b/setup.py @@ -1,2 +1,4 @@ +from __future__ import annotations + from setuptools import setup setup() diff --git a/testing/auto_namedtuple.py b/testing/auto_namedtuple.py index 02e08fef..d5a43775 100644 --- a/testing/auto_namedtuple.py +++ b/testing/auto_namedtuple.py @@ -1,4 +1,4 @@ -from __future__ import unicode_literals +from __future__ import annotations import collections diff --git a/testing/fixtures.py b/testing/fixtures.py index 70d0750d..79a11605 100644 --- a/testing/fixtures.py +++ b/testing/fixtures.py @@ -1,13 +1,9 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import contextlib -import io import os.path import shutil -from aspy.yaml import ordered_dump -from aspy.yaml import ordered_load from cfgv import apply_defaults from cfgv import validate @@ -16,6 +12,8 @@ from pre_commit import git from pre_commit.clientlib import CONFIG_SCHEMA from pre_commit.clientlib import load_manifest from pre_commit.util import cmd_output +from pre_commit.yaml import yaml_dump +from pre_commit.yaml import yaml_load from testing.util import get_resource_path from testing.util import git_commit @@ -40,7 +38,7 @@ def copy_tree_to_path(src_dir, dest_dir): def git_dir(tempdir_factory): path = tempdir_factory.get() - cmd_output('git', 'init', path) + cmd_output('git', '-c', 'init.defaultBranch=master', 'init', path) return path @@ -58,11 +56,11 @@ def modify_manifest(path, commit=True): .pre-commit-hooks.yaml. """ manifest_path = os.path.join(path, C.MANIFEST_FILE) - with io.open(manifest_path) as f: - manifest = ordered_load(f.read()) + with open(manifest_path) as f: + manifest = yaml_load(f.read()) yield manifest - with io.open(manifest_path, 'w') as manifest_file: - manifest_file.write(ordered_dump(manifest, **C.YAML_DUMP_KWARGS)) + with open(manifest_path, 'w') as manifest_file: + manifest_file.write(yaml_dump(manifest)) if commit: git_commit(msg=modify_manifest.__name__, cwd=path) @@ -73,11 +71,11 @@ def modify_config(path='.', commit=True): .pre-commit-config.yaml """ config_path = os.path.join(path, C.CONFIG_FILE) - with io.open(config_path) as f: - config = ordered_load(f.read()) + with open(config_path) as f: + config = yaml_load(f.read()) yield config - with io.open(config_path, 'w', encoding='UTF-8') as config_file: - config_file.write(ordered_dump(config, **C.YAML_DUMP_KWARGS)) + with open(config_path, 'w', encoding='UTF-8') as config_file: + config_file.write(yaml_dump(config)) if commit: git_commit(msg=modify_config.__name__, cwd=path) @@ -101,7 +99,7 @@ def sample_meta_config(): def make_config_from_repo(repo_path, rev=None, hooks=None, check=True): manifest = load_manifest(os.path.join(repo_path, C.MANIFEST_FILE)) config = { - 'repo': 'file://{}'.format(repo_path), + 'repo': f'file://{repo_path}', 'rev': rev or git.head_rev(repo_path), 'hooks': hooks or [{'id': hook['id']} for hook in manifest], } @@ -117,8 +115,8 @@ def make_config_from_repo(repo_path, rev=None, hooks=None, check=True): def read_config(directory, config_file=C.CONFIG_FILE): config_path = os.path.join(directory, config_file) - with io.open(config_path) as f: - config = ordered_load(f.read()) + with open(config_path) as f: + config = yaml_load(f.read()) return config @@ -126,8 +124,8 @@ def write_config(directory, config, config_file=C.CONFIG_FILE): if type(config) is not list and 'repos' not in config: assert isinstance(config, dict), config config = {'repos': [config]} - with io.open(os.path.join(directory, config_file), 'w') as outfile: - outfile.write(ordered_dump(config, **C.YAML_DUMP_KWARGS)) + with open(os.path.join(directory, config_file), 'w') as outfile: + outfile.write(yaml_dump(config)) def add_config_to_repo(git_path, config, config_file=C.CONFIG_FILE): diff --git a/testing/get-coursier.sh b/testing/get-coursier.sh new file mode 100755 index 00000000..958e73b2 --- /dev/null +++ b/testing/get-coursier.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [ "$OSTYPE" = msys ]; then + URL='https://github.com/coursier/coursier/releases/download/v2.1.0-RC4/cs-x86_64-pc-win32.zip' + SHA256='0d07386ff0f337e3e6264f7dde29d137dda6eaa2385f29741435e0b93ccdb49d' + TARGET='/tmp/coursier/cs.zip' + + unpack() { + unzip "$TARGET" -d /tmp/coursier + mv /tmp/coursier/cs-*.exe /tmp/coursier/cs.exe + cygpath -w /tmp/coursier >> "$GITHUB_PATH" + } +else + URL='https://github.com/coursier/coursier/releases/download/v2.1.0-RC4/cs-x86_64-pc-linux.gz' + SHA256='176e92e08ab292531aa0c4993dbc9f2c99dec79578752f3b9285f54f306db572' + TARGET=/tmp/coursier/cs.gz + + unpack() { + gunzip "$TARGET" + chmod +x /tmp/coursier/cs + echo /tmp/coursier >> "$GITHUB_PATH" + } +fi + +mkdir -p /tmp/coursier +curl --location --silent --output "$TARGET" "$URL" +echo "$SHA256 $TARGET" | sha256sum --check +unpack diff --git a/testing/get-dart.sh b/testing/get-dart.sh new file mode 100755 index 00000000..b4545e71 --- /dev/null +++ b/testing/get-dart.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail + +VERSION=2.19.6 + +if [ "$OSTYPE" = msys ]; then + URL="https://storage.googleapis.com/dart-archive/channels/stable/release/${VERSION}/sdk/dartsdk-windows-x64-release.zip" + cygpath -w /tmp/dart-sdk/bin >> "$GITHUB_PATH" +else + URL="https://storage.googleapis.com/dart-archive/channels/stable/release/${VERSION}/sdk/dartsdk-linux-x64-release.zip" + echo '/tmp/dart-sdk/bin' >> "$GITHUB_PATH" +fi + +curl --silent --location --output /tmp/dart.zip "$URL" + +unzip -q -d /tmp /tmp/dart.zip +rm /tmp/dart.zip diff --git a/testing/get-swift.sh b/testing/get-swift.sh deleted file mode 100755 index 28986a5f..00000000 --- a/testing/get-swift.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env bash -# This is a script used in travis-ci to install swift -set -euxo pipefail - -. /etc/lsb-release -if [ "$DISTRIB_CODENAME" = "trusty" ]; then - SWIFT_URL='https://swift.org/builds/swift-4.0.3-release/ubuntu1404/swift-4.0.3-RELEASE/swift-4.0.3-RELEASE-ubuntu14.04.tar.gz' - SWIFT_HASH="dddb40ec4956e4f6a3f4532d859691d5d1ba8822f6e8b4ec6c452172dbede5ae" -else - SWIFT_URL='https://swift.org/builds/swift-4.0.3-release/ubuntu1604/swift-4.0.3-RELEASE/swift-4.0.3-RELEASE-ubuntu16.04.tar.gz' - SWIFT_HASH="9adf64cabc7c02ea2d08f150b449b05e46bd42d6e542bf742b3674f5c37f0dbf" -fi - -check() { - echo "$SWIFT_HASH $TGZ" | sha256sum --check -} - -TGZ="$HOME/.swift/swift.tar.gz" -mkdir -p "$(dirname "$TGZ")" -if ! check >& /dev/null; then - rm -f "$TGZ" - curl --location --silent --output "$TGZ" "$SWIFT_URL" - check -fi - -mkdir -p /tmp/swift -tar -xf "$TGZ" --strip 1 --directory /tmp/swift diff --git a/testing/language_helpers.py b/testing/language_helpers.py new file mode 100644 index 00000000..05c94ebc --- /dev/null +++ b/testing/language_helpers.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +import os +from collections.abc import Sequence + +from pre_commit.lang_base import Language +from pre_commit.prefix import Prefix + + +def run_language( + path: os.PathLike[str], + language: Language, + exe: str, + args: Sequence[str] = (), + file_args: Sequence[str] = (), + version: str | None = None, + deps: Sequence[str] = (), + is_local: bool = False, + require_serial: bool = True, + color: bool = False, +) -> tuple[int, bytes]: + prefix = Prefix(str(path)) + version = version or language.get_default_version() + + if language.ENVIRONMENT_DIR is not None: + language.install_environment(prefix, version, deps) + health_error = language.health_check(prefix, version) + assert health_error is None, health_error + with language.in_env(prefix, version): + ret, out = language.run_hook( + prefix, + exe, + args, + file_args, + is_local=is_local, + require_serial=require_serial, + color=color, + ) + out = out.replace(b'\r\n', b'\n') + return ret, out diff --git a/testing/languages b/testing/languages new file mode 100755 index 00000000..f4804c7e --- /dev/null +++ b/testing/languages @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import concurrent.futures +import json +import os.path +import subprocess +import sys + +EXCLUDED = frozenset(( + ('windows-latest', 'docker'), + ('windows-latest', 'docker_image'), + ('windows-latest', 'lua'), + ('windows-latest', 'swift'), +)) + + +def _always_run() -> frozenset[str]: + ret = ['.github/workflows/languages.yaml', 'testing/languages'] + ret.extend( + os.path.join('pre_commit/resources', fname) + for fname in os.listdir('pre_commit/resources') + ) + return frozenset(ret) + + +def _lang_files(lang: str) -> frozenset[str]: + prog = f'''\ +import json +import os.path +import sys + +import pre_commit.languages.{lang} +import tests.languages.{lang}_test + +modules = sorted( + os.path.relpath(v.__file__) + for k, v in sys.modules.items() + if k.startswith(('pre_commit.', 'tests.', 'testing.')) +) +print(json.dumps(modules)) +''' + out = json.loads(subprocess.check_output((sys.executable, '-c', prog))) + return frozenset(out) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument('--all', action='store_true') + args = parser.parse_args() + + langs = [ + os.path.splitext(fname)[0] + for fname in sorted(os.listdir('pre_commit/languages')) + if fname.endswith('.py') and fname != '__init__.py' + ] + + triggers_all = _always_run() + for fname in triggers_all: + assert os.path.exists(fname), fname + + if not args.all: + with concurrent.futures.ThreadPoolExecutor(os.cpu_count()) as exe: + by_lang = { + lang: files | triggers_all + for lang, files in zip(langs, exe.map(_lang_files, langs)) + } + + diff_cmd = ('git', 'diff', '--name-only', 'origin/main...HEAD') + files = set(subprocess.check_output(diff_cmd).decode().splitlines()) + + langs = [ + lang + for lang, lang_files in by_lang.items() + if lang_files & files + ] + + matched = [ + {'os': os, 'language': lang} + for os in ('windows-latest', 'ubuntu-latest') + for lang in langs + if (os, lang) not in EXCLUDED + ] + + with open(os.environ['GITHUB_OUTPUT'], 'a') as f: + f.write(f'languages={json.dumps(matched)}\n') + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/testing/latest-git.sh b/testing/latest-git.sh deleted file mode 100755 index 0f7a52a6..00000000 --- a/testing/latest-git.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env bash -# This is a script used in travis-ci to have latest git -set -ex -git clone git://github.com/git/git --depth 1 /tmp/git -pushd /tmp/git -make prefix=/tmp/git -j8 install -popd diff --git a/testing/make-archives b/testing/make-archives new file mode 100755 index 00000000..10f40a3a --- /dev/null +++ b/testing/make-archives @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import gzip +import os.path +import shutil +import subprocess +import tarfile +import tempfile +from collections.abc import Sequence + + +# This is a script for generating the tarred resources for git repo +# dependencies. Currently it's just for "vendoring" ruby support packages. + + +REPOS = ( + ('rbenv', 'https://github.com/rbenv/rbenv', '10e96bfc'), + ('ruby-build', 'https://github.com/rbenv/ruby-build', '447468b1'), + ( + 'ruby-download', + 'https://github.com/garnieretienne/rvm-download', + '09bd7c6', + ), +) + + +def reset(tarinfo: tarfile.TarInfo) -> tarfile.TarInfo: + tarinfo.uid = tarinfo.gid = 0 + tarinfo.uname = tarinfo.gname = 'root' + tarinfo.mtime = 0 + return tarinfo + + +def make_archive(name: str, repo: str, ref: str, destdir: str) -> str: + output_path = os.path.join(destdir, f'{name}.tar.gz') + with tempfile.TemporaryDirectory() as tmpdir: + # this ensures that the root directory has umask permissions + gitdir = os.path.join(tmpdir, 'root') + + # Clone the repository to the temporary directory + subprocess.check_call(('git', 'clone', repo, gitdir)) + subprocess.check_call(('git', '-C', gitdir, 'checkout', ref)) + + # We don't want the '.git' directory + # It adds a bunch of size to the archive and we don't use it at + # runtime + shutil.rmtree(os.path.join(gitdir, '.git')) + + arcs = [(name, gitdir)] + for root, dirs, filenames in os.walk(gitdir): + for filename in dirs + filenames: + abspath = os.path.abspath(os.path.join(root, filename)) + relpath = os.path.relpath(abspath, gitdir) + arcs.append((os.path.join(name, relpath), abspath)) + arcs.sort() + + with gzip.GzipFile(output_path, 'wb', mtime=0) as gzipf: + with tarfile.open(fileobj=gzipf, mode='w') as tf: + for arcname, abspath in arcs: + tf.add( + abspath, + arcname=arcname, + recursive=False, + filter=reset, + ) + + return output_path + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser() + parser.add_argument('--dest', default='pre_commit/resources') + args = parser.parse_args(argv) + for archive_name, repo, ref in REPOS: + print(f'Making {archive_name}.tar.gz for {repo}@{ref}') + make_archive(archive_name, repo, ref, args.dest) + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/testing/resources/arbitrary_bytes_repo/.pre-commit-hooks.yaml b/testing/resources/arbitrary_bytes_repo/.pre-commit-hooks.yaml index 2c237009..c2aec9b9 100644 --- a/testing/resources/arbitrary_bytes_repo/.pre-commit-hooks.yaml +++ b/testing/resources/arbitrary_bytes_repo/.pre-commit-hooks.yaml @@ -1,6 +1,5 @@ -- id: python3-hook - name: Python 3 Hook - entry: python3-hook - language: python - language_version: python3 +- id: hook + name: hook + entry: ./hook.sh + language: script files: \.py$ diff --git a/testing/resources/arbitrary_bytes_repo/hook.sh b/testing/resources/arbitrary_bytes_repo/hook.sh new file mode 100755 index 00000000..9df0c5a0 --- /dev/null +++ b/testing/resources/arbitrary_bytes_repo/hook.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# Intentionally write mixed encoding to the output. This should not crash +# pre-commit and should write bytes to the output. +# 'β˜ƒ'.encode() + 'Β²'.encode('latin1') +echo -e '\xe2\x98\x83\xb2' +# exit 1 to trigger printing +exit 1 diff --git a/testing/resources/arbitrary_bytes_repo/python3_hook.py b/testing/resources/arbitrary_bytes_repo/python3_hook.py deleted file mode 100644 index ba698a93..00000000 --- a/testing/resources/arbitrary_bytes_repo/python3_hook.py +++ /dev/null @@ -1,13 +0,0 @@ -# -*- coding: utf-8 -*- -from __future__ import print_function -from __future__ import unicode_literals - -import sys - - -def main(): - # Intentionally write mixed encoding to the output. This should not crash - # pre-commit and should write bytes to the output. - sys.stdout.buffer.write('β˜ƒ'.encode('UTF-8') + 'Β²'.encode('latin1') + b'\n') - # Return 1 to trigger printing - return 1 diff --git a/testing/resources/arbitrary_bytes_repo/setup.py b/testing/resources/arbitrary_bytes_repo/setup.py deleted file mode 100644 index c780e427..00000000 --- a/testing/resources/arbitrary_bytes_repo/setup.py +++ /dev/null @@ -1,8 +0,0 @@ -from setuptools import setup - -setup( - name='python3_hook', - version='0.0.0', - py_modules=['python3_hook'], - entry_points={'console_scripts': ['python3-hook=python3_hook:main']}, -) diff --git a/testing/resources/docker_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/docker_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index 52957396..00000000 --- a/testing/resources/docker_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,17 +0,0 @@ -- id: docker-hook - name: Docker test hook - entry: echo - language: docker - files: \.txt$ - -- id: docker-hook-arg - name: Docker test hook - entry: echo -n - language: docker - files: \.txt$ - -- id: docker-hook-failing - name: Docker test hook with nonzero exit code - entry: bork - language: docker - files: \.txt$ diff --git a/testing/resources/docker_hooks_repo/Dockerfile b/testing/resources/docker_hooks_repo/Dockerfile deleted file mode 100644 index 841b151b..00000000 --- a/testing/resources/docker_hooks_repo/Dockerfile +++ /dev/null @@ -1,3 +0,0 @@ -FROM cogniteev/echo - -CMD ["echo", "This is overwritten by the .pre-commit-hooks.yaml 'entry'"] diff --git a/testing/resources/docker_image_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/docker_image_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index 1b385aa1..00000000 --- a/testing/resources/docker_image_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,8 +0,0 @@ -- id: echo-entrypoint - name: echo (via --entrypoint) - language: docker_image - entry: --entrypoint echo cogniteev/echo -- id: echo-cmd - name: echo (via cmd) - language: docker_image - entry: cogniteev/echo echo diff --git a/testing/resources/exclude_types_repo/bin/hook.sh b/testing/resources/exclude_types_repo/bin/hook.sh index bdade513..a828db4d 100755 --- a/testing/resources/exclude_types_repo/bin/hook.sh +++ b/testing/resources/exclude_types_repo/bin/hook.sh @@ -1,3 +1,3 @@ #!/usr/bin/env bash -echo $@ +echo "$@" exit 1 diff --git a/testing/resources/failing_hook_repo/bin/hook.sh b/testing/resources/failing_hook_repo/bin/hook.sh index 229ccaf4..7dcffebe 100755 --- a/testing/resources/failing_hook_repo/bin/hook.sh +++ b/testing/resources/failing_hook_repo/bin/hook.sh @@ -1,4 +1,4 @@ #!/usr/bin/env bash echo 'Fail' -echo $@ +echo "$@" exit 1 diff --git a/testing/resources/golang_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/golang_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index 206733bb..00000000 --- a/testing/resources/golang_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,5 +0,0 @@ -- id: golang-hook - name: golang example hook - entry: golang-hello-world - language: golang - files: '' diff --git a/testing/resources/golang_hooks_repo/golang-hello-world/main.go b/testing/resources/golang_hooks_repo/golang-hello-world/main.go deleted file mode 100644 index 1e3c591a..00000000 --- a/testing/resources/golang_hooks_repo/golang-hello-world/main.go +++ /dev/null @@ -1,17 +0,0 @@ -package main - - -import ( - "fmt" - "github.com/BurntSushi/toml" -) - -type Config struct { - What string -} - -func main() { - var conf Config - toml.Decode("What = 'world'\n", &conf) - fmt.Printf("hello %v\n", conf.What) -} diff --git a/testing/resources/modified_file_returns_zero_repo/bin/hook2.sh b/testing/resources/modified_file_returns_zero_repo/bin/hook2.sh index 5af177a8..a9f1dcd9 100755 --- a/testing/resources/modified_file_returns_zero_repo/bin/hook2.sh +++ b/testing/resources/modified_file_returns_zero_repo/bin/hook2.sh @@ -1,2 +1,2 @@ #!/usr/bin/env bash -echo $@ +echo "$@" diff --git a/testing/resources/node_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/node_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index 257698a4..00000000 --- a/testing/resources/node_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,5 +0,0 @@ -- id: foo - name: Foo - entry: foo - language: node - files: \.js$ diff --git a/testing/resources/node_hooks_repo/bin/main.js b/testing/resources/node_hooks_repo/bin/main.js deleted file mode 100644 index 8e0f025a..00000000 --- a/testing/resources/node_hooks_repo/bin/main.js +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env node - -console.log('Hello World'); diff --git a/testing/resources/node_hooks_repo/package.json b/testing/resources/node_hooks_repo/package.json deleted file mode 100644 index 050b6300..00000000 --- a/testing/resources/node_hooks_repo/package.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "foo", - "version": "0.0.1", - "bin": {"foo": "./bin/main.js"} -} diff --git a/testing/resources/node_versioned_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/node_versioned_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index e7ad5ea7..00000000 --- a/testing/resources/node_versioned_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,6 +0,0 @@ -- id: versioned-node-hook - name: Versioned node hook - entry: versioned-node-hook - language: node - language_version: 9.3.0 - files: \.js$ diff --git a/testing/resources/node_versioned_hooks_repo/bin/main.js b/testing/resources/node_versioned_hooks_repo/bin/main.js deleted file mode 100644 index df12cbeb..00000000 --- a/testing/resources/node_versioned_hooks_repo/bin/main.js +++ /dev/null @@ -1,4 +0,0 @@ -#!/usr/bin/env node - -console.log(process.version); -console.log('Hello World'); diff --git a/testing/resources/node_versioned_hooks_repo/package.json b/testing/resources/node_versioned_hooks_repo/package.json deleted file mode 100644 index 18c7787c..00000000 --- a/testing/resources/node_versioned_hooks_repo/package.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "versioned-node-hook", - "version": "0.0.1", - "bin": {"versioned-node-hook": "./bin/main.js"} -} diff --git a/testing/resources/python3_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/python3_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index 2c237009..00000000 --- a/testing/resources/python3_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,6 +0,0 @@ -- id: python3-hook - name: Python 3 Hook - entry: python3-hook - language: python - language_version: python3 - files: \.py$ diff --git a/testing/resources/python3_hooks_repo/py3_hook.py b/testing/resources/python3_hooks_repo/py3_hook.py deleted file mode 100644 index f0f88088..00000000 --- a/testing/resources/python3_hooks_repo/py3_hook.py +++ /dev/null @@ -1,10 +0,0 @@ -from __future__ import print_function - -import sys - - -def main(): - print(sys.version_info[0]) - print(repr(sys.argv[1:])) - print('Hello World') - return 0 diff --git a/testing/resources/python3_hooks_repo/setup.py b/testing/resources/python3_hooks_repo/setup.py deleted file mode 100644 index 9125dc1d..00000000 --- a/testing/resources/python3_hooks_repo/setup.py +++ /dev/null @@ -1,8 +0,0 @@ -from setuptools import setup - -setup( - name='python3_hook', - version='0.0.0', - py_modules=['py3_hook'], - entry_points={'console_scripts': ['python3-hook = py3_hook:main']}, -) diff --git a/testing/resources/python_hooks_repo/foo.py b/testing/resources/python_hooks_repo/foo.py index 412a5c62..40efde39 100644 --- a/testing/resources/python_hooks_repo/foo.py +++ b/testing/resources/python_hooks_repo/foo.py @@ -1,4 +1,4 @@ -from __future__ import print_function +from __future__ import annotations import sys diff --git a/testing/resources/python_hooks_repo/setup.py b/testing/resources/python_hooks_repo/setup.py index 0559271e..cff6cadf 100644 --- a/testing/resources/python_hooks_repo/setup.py +++ b/testing/resources/python_hooks_repo/setup.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from setuptools import setup setup( diff --git a/testing/resources/python_venv_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/python_venv_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index a666ed87..00000000 --- a/testing/resources/python_venv_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,5 +0,0 @@ -- id: foo - name: Foo - entry: foo - language: python_venv - files: \.py$ diff --git a/testing/resources/python_venv_hooks_repo/foo.py b/testing/resources/python_venv_hooks_repo/foo.py deleted file mode 100644 index 412a5c62..00000000 --- a/testing/resources/python_venv_hooks_repo/foo.py +++ /dev/null @@ -1,9 +0,0 @@ -from __future__ import print_function - -import sys - - -def main(): - print(repr(sys.argv[1:])) - print('Hello World') - return 0 diff --git a/testing/resources/python_venv_hooks_repo/foo/__init__.py b/testing/resources/python_venv_hooks_repo/foo/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/testing/resources/python_venv_hooks_repo/setup.py b/testing/resources/python_venv_hooks_repo/setup.py deleted file mode 100644 index 0559271e..00000000 --- a/testing/resources/python_venv_hooks_repo/setup.py +++ /dev/null @@ -1,8 +0,0 @@ -from setuptools import setup - -setup( - name='foo', - version='0.0.0', - py_modules=['foo'], - entry_points={'console_scripts': ['foo = foo:main']}, -) diff --git a/testing/resources/ruby_hooks_repo/.gitignore b/testing/resources/ruby_hooks_repo/.gitignore deleted file mode 100644 index c111b331..00000000 --- a/testing/resources/ruby_hooks_repo/.gitignore +++ /dev/null @@ -1 +0,0 @@ -*.gem diff --git a/testing/resources/ruby_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/ruby_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index aa15872f..00000000 --- a/testing/resources/ruby_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,5 +0,0 @@ -- id: ruby_hook - name: Ruby Hook - entry: ruby_hook - language: ruby - files: \.rb$ diff --git a/testing/resources/ruby_hooks_repo/bin/ruby_hook b/testing/resources/ruby_hooks_repo/bin/ruby_hook deleted file mode 100755 index 5a7e5ed2..00000000 --- a/testing/resources/ruby_hooks_repo/bin/ruby_hook +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env ruby - -puts 'Hello world from a ruby hook' diff --git a/testing/resources/ruby_hooks_repo/lib/.gitignore b/testing/resources/ruby_hooks_repo/lib/.gitignore deleted file mode 100644 index e69de29b..00000000 diff --git a/testing/resources/ruby_hooks_repo/ruby_hook.gemspec b/testing/resources/ruby_hooks_repo/ruby_hook.gemspec deleted file mode 100644 index 75f4e8f7..00000000 --- a/testing/resources/ruby_hooks_repo/ruby_hook.gemspec +++ /dev/null @@ -1,9 +0,0 @@ -Gem::Specification.new do |s| - s.name = 'ruby_hook' - s.version = '0.1.0' - s.authors = ['Anthony Sottile'] - s.summary = 'A ruby hook!' - s.description = 'A ruby hook!' - s.files = ['bin/ruby_hook'] - s.executables = ['ruby_hook'] -end diff --git a/testing/resources/ruby_versioned_hooks_repo/.gitignore b/testing/resources/ruby_versioned_hooks_repo/.gitignore deleted file mode 100644 index c111b331..00000000 --- a/testing/resources/ruby_versioned_hooks_repo/.gitignore +++ /dev/null @@ -1 +0,0 @@ -*.gem diff --git a/testing/resources/ruby_versioned_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/ruby_versioned_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index 63e1dd4c..00000000 --- a/testing/resources/ruby_versioned_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,6 +0,0 @@ -- id: ruby_hook - name: Ruby Hook - entry: ruby_hook - language: ruby - language_version: 2.5.1 - files: \.rb$ diff --git a/testing/resources/ruby_versioned_hooks_repo/bin/ruby_hook b/testing/resources/ruby_versioned_hooks_repo/bin/ruby_hook deleted file mode 100755 index 2406f04c..00000000 --- a/testing/resources/ruby_versioned_hooks_repo/bin/ruby_hook +++ /dev/null @@ -1,4 +0,0 @@ -#!/usr/bin/env ruby - -puts RUBY_VERSION -puts 'Hello world from a ruby hook' diff --git a/testing/resources/ruby_versioned_hooks_repo/lib/.gitignore b/testing/resources/ruby_versioned_hooks_repo/lib/.gitignore deleted file mode 100644 index e69de29b..00000000 diff --git a/testing/resources/ruby_versioned_hooks_repo/ruby_hook.gemspec b/testing/resources/ruby_versioned_hooks_repo/ruby_hook.gemspec deleted file mode 100644 index 75f4e8f7..00000000 --- a/testing/resources/ruby_versioned_hooks_repo/ruby_hook.gemspec +++ /dev/null @@ -1,9 +0,0 @@ -Gem::Specification.new do |s| - s.name = 'ruby_hook' - s.version = '0.1.0' - s.authors = ['Anthony Sottile'] - s.summary = 'A ruby hook!' - s.description = 'A ruby hook!' - s.files = ['bin/ruby_hook'] - s.executables = ['ruby_hook'] -end diff --git a/testing/resources/rust_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/rust_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index df1269ff..00000000 --- a/testing/resources/rust_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,5 +0,0 @@ -- id: rust-hook - name: rust example hook - entry: rust-hello-world - language: rust - files: '' diff --git a/testing/resources/rust_hooks_repo/Cargo.lock b/testing/resources/rust_hooks_repo/Cargo.lock deleted file mode 100644 index 36fbfda2..00000000 --- a/testing/resources/rust_hooks_repo/Cargo.lock +++ /dev/null @@ -1,3 +0,0 @@ -[[package]] -name = "rust-hello-world" -version = "0.1.0" diff --git a/testing/resources/rust_hooks_repo/Cargo.toml b/testing/resources/rust_hooks_repo/Cargo.toml deleted file mode 100644 index cd83b435..00000000 --- a/testing/resources/rust_hooks_repo/Cargo.toml +++ /dev/null @@ -1,3 +0,0 @@ -[package] -name = "rust-hello-world" -version = "0.1.0" diff --git a/testing/resources/rust_hooks_repo/src/main.rs b/testing/resources/rust_hooks_repo/src/main.rs deleted file mode 100644 index ad379d6e..00000000 --- a/testing/resources/rust_hooks_repo/src/main.rs +++ /dev/null @@ -1,3 +0,0 @@ -fn main() { - println!("hello world"); -} diff --git a/testing/resources/script_hooks_repo/bin/hook.sh b/testing/resources/script_hooks_repo/bin/hook.sh index 6565ee40..cbc4b354 100755 --- a/testing/resources/script_hooks_repo/bin/hook.sh +++ b/testing/resources/script_hooks_repo/bin/hook.sh @@ -1,4 +1,4 @@ #!/usr/bin/env bash -echo $@ +echo "$@" echo 'Hello World' diff --git a/testing/resources/stdout_stderr_repo/.pre-commit-hooks.yaml b/testing/resources/stdout_stderr_repo/.pre-commit-hooks.yaml new file mode 100644 index 00000000..6800d259 --- /dev/null +++ b/testing/resources/stdout_stderr_repo/.pre-commit-hooks.yaml @@ -0,0 +1,8 @@ +- id: stdout-stderr + name: stdout-stderr + language: script + entry: ./stdout-stderr-entry +- id: tty-check + name: tty-check + language: script + entry: ./tty-check-entry diff --git a/testing/resources/stdout_stderr_repo/stdout-stderr-entry b/testing/resources/stdout_stderr_repo/stdout-stderr-entry new file mode 100755 index 00000000..7563df53 --- /dev/null +++ b/testing/resources/stdout_stderr_repo/stdout-stderr-entry @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +echo 0 +echo 1 1>&2 +echo 2 +echo 3 1>&2 +echo 4 +echo 5 1>&2 diff --git a/testing/resources/stdout_stderr_repo/tty-check-entry b/testing/resources/stdout_stderr_repo/tty-check-entry new file mode 100755 index 00000000..01a9d388 --- /dev/null +++ b/testing/resources/stdout_stderr_repo/tty-check-entry @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +t() { + if [ -t "$1" ]; then + echo "$2: True" + else + echo "$2: False" + fi +} +t 0 stdin +t 1 stdout +t 2 stderr diff --git a/testing/resources/swift_hooks_repo/.gitignore b/testing/resources/swift_hooks_repo/.gitignore deleted file mode 100644 index 02c08753..00000000 --- a/testing/resources/swift_hooks_repo/.gitignore +++ /dev/null @@ -1,4 +0,0 @@ -.DS_Store -/.build -/Packages -/*.xcodeproj diff --git a/testing/resources/swift_hooks_repo/.pre-commit-hooks.yaml b/testing/resources/swift_hooks_repo/.pre-commit-hooks.yaml deleted file mode 100644 index c08df87d..00000000 --- a/testing/resources/swift_hooks_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,6 +0,0 @@ -- id: swift-hooks-repo - name: Swift hooks repo example - description: Runs the hello world app generated by swift package init --type executable (binary called swift_hooks_repo here) - entry: swift_hooks_repo - language: swift - files: \.(swift)$ diff --git a/testing/resources/swift_hooks_repo/Package.swift b/testing/resources/swift_hooks_repo/Package.swift deleted file mode 100644 index 6e02c188..00000000 --- a/testing/resources/swift_hooks_repo/Package.swift +++ /dev/null @@ -1,5 +0,0 @@ -import PackageDescription - -let package = Package( - name: "swift_hooks_repo" -) diff --git a/testing/resources/swift_hooks_repo/Sources/main.swift b/testing/resources/swift_hooks_repo/Sources/main.swift deleted file mode 100644 index f7cf60e1..00000000 --- a/testing/resources/swift_hooks_repo/Sources/main.swift +++ /dev/null @@ -1 +0,0 @@ -print("Hello, world!") diff --git a/testing/resources/system_hook_with_spaces_repo/.pre-commit-hooks.yaml b/testing/resources/system_hook_with_spaces_repo/.pre-commit-hooks.yaml deleted file mode 100644 index b2c347c1..00000000 --- a/testing/resources/system_hook_with_spaces_repo/.pre-commit-hooks.yaml +++ /dev/null @@ -1,5 +0,0 @@ -- id: system-hook-with-spaces - name: System hook with spaces - entry: bash -c 'echo "Hello World"' - language: system - files: \.sh$ diff --git a/testing/resources/types_or_repo/.pre-commit-hooks.yaml b/testing/resources/types_or_repo/.pre-commit-hooks.yaml new file mode 100644 index 00000000..a4ea920d --- /dev/null +++ b/testing/resources/types_or_repo/.pre-commit-hooks.yaml @@ -0,0 +1,6 @@ +- id: python-cython-files + name: Python and Cython files + entry: bin/hook.sh + language: script + types: [file] + types_or: [python, cython] diff --git a/testing/resources/types_or_repo/bin/hook.sh b/testing/resources/types_or_repo/bin/hook.sh new file mode 100755 index 00000000..a828db4d --- /dev/null +++ b/testing/resources/types_or_repo/bin/hook.sh @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +echo "$@" +exit 1 diff --git a/testing/resources/types_repo/bin/hook.sh b/testing/resources/types_repo/bin/hook.sh index bdade513..a828db4d 100755 --- a/testing/resources/types_repo/bin/hook.sh +++ b/testing/resources/types_repo/bin/hook.sh @@ -1,3 +1,3 @@ #!/usr/bin/env bash -echo $@ +echo "$@" exit 1 diff --git a/testing/util.py b/testing/util.py index d82612fa..1646ccd2 100644 --- a/testing/util.py +++ b/testing/util.py @@ -1,14 +1,12 @@ -from __future__ import unicode_literals +from __future__ import annotations import contextlib import os.path +import subprocess import sys import pytest -from pre_commit import parse_shebang -from pre_commit.languages.docker import docker_is_running -from pre_commit.languages.pcre import GREP from pre_commit.util import cmd_output from testing.auto_namedtuple import auto_namedtuple @@ -20,80 +18,20 @@ def get_resource_path(path): return os.path.join(TESTING_DIR, 'resources', path) -def cmd_output_mocked_pre_commit_home(*args, **kwargs): - # keyword-only argument - tempdir_factory = kwargs.pop('tempdir_factory') - pre_commit_home = kwargs.pop('pre_commit_home', tempdir_factory.get()) +def cmd_output_mocked_pre_commit_home( + *args, tempdir_factory, pre_commit_home=None, env=None, **kwargs, +): + if pre_commit_home is None: + pre_commit_home = tempdir_factory.get() + env = env if env is not None else os.environ + kwargs.setdefault('stderr', subprocess.STDOUT) # Don't want to write to the home directory - env = dict(kwargs.pop('env', os.environ), PRE_COMMIT_HOME=pre_commit_home) - return cmd_output(*args, env=env, **kwargs) + env = dict(env, PRE_COMMIT_HOME=pre_commit_home) + ret, out, _ = cmd_output(*args, env=env, **kwargs) + return ret, out.replace('\r\n', '\n'), None -skipif_cant_run_docker = pytest.mark.skipif( - os.name == 'nt' or not docker_is_running(), - reason="Docker isn't running or can't be accessed", -) - -skipif_cant_run_swift = pytest.mark.skipif( - parse_shebang.find_executable('swift') is None, - reason='swift isn\'t installed or can\'t be found', -) - -xfailif_windows_no_ruby = pytest.mark.xfail( - os.name == 'nt', - reason='Ruby support not yet implemented on windows.', -) - - -def broken_deep_listdir(): # pragma: no cover (platform specific) - if sys.platform != 'win32': - return False - try: - os.listdir(str('\\\\?\\') + os.path.abspath(str('.'))) - except OSError: - return True - try: - os.listdir(b'\\\\?\\C:' + b'\\' * 300) - except TypeError: - return True - except OSError: - return False - - -xfailif_broken_deep_listdir = pytest.mark.xfail( - broken_deep_listdir(), - reason='Node on windows requires deep listdir', -) - - -def platform_supports_pcre(): - output = cmd_output(GREP, '-P', "Don't", 'CHANGELOG.md', retcode=None) - return output[0] == 0 and "Don't use readlink -f" in output[1] - - -xfailif_no_pcre_support = pytest.mark.xfail( - not platform_supports_pcre(), - reason='grep -P is not supported on this platform', -) - -xfailif_no_symlink = pytest.mark.xfail( - not hasattr(os, 'symlink'), - reason='Symlink is not supported on this platform', -) - - -def supports_venv(): # pragma: no cover (platform specific) - try: - __import__('ensurepip') - __import__('venv') - return True - except ImportError: - return False - - -xfailif_no_venv = pytest.mark.xfail( - not supports_venv(), reason='Does not support venv module', -) +xfailif_windows = pytest.mark.xfail(sys.platform == 'win32', reason='windows') def run_opts( @@ -102,11 +40,23 @@ def run_opts( color=False, verbose=False, hook=None, - origin='', - source='', - hook_stage='commit', + fail_fast=False, + remote_branch='', + local_branch='', + from_ref='', + to_ref='', + pre_rebase_upstream='', + pre_rebase_branch='', + remote_name='', + remote_url='', + hook_stage='pre-commit', show_diff_on_failure=False, commit_msg_filename='', + prepare_commit_message_source='', + commit_object_name='', + checkout_type='', + is_squash_merge='', + rewrite_command='', ): # These are mutually exclusive assert not (all_files and files) @@ -116,11 +66,23 @@ def run_opts( color=color, verbose=verbose, hook=hook, - origin=origin, - source=source, + fail_fast=fail_fast, + remote_branch=remote_branch, + local_branch=local_branch, + from_ref=from_ref, + to_ref=to_ref, + pre_rebase_upstream=pre_rebase_upstream, + pre_rebase_branch=pre_rebase_branch, + remote_name=remote_name, + remote_url=remote_url, hook_stage=hook_stage, show_diff_on_failure=show_diff_on_failure, commit_msg_filename=commit_msg_filename, + prepare_commit_message_source=prepare_commit_message_source, + commit_object_name=commit_object_name, + checkout_type=checkout_type, + is_squash_merge=is_squash_merge, + rewrite_command=rewrite_command, ) @@ -134,11 +96,13 @@ def cwd(path): os.chdir(original_cwd) -def git_commit(*args, **kwargs): - fn = kwargs.pop('fn', cmd_output) - msg = kwargs.pop('msg', 'commit!') +def git_commit(*args, fn=cmd_output, msg='commit!', all_files=True, **kwargs): + kwargs.setdefault('stderr', subprocess.STDOUT) - cmd = ('git', 'commit', '--allow-empty', '--no-gpg-sign', '-a') + args + cmd = ('git', 'commit', '--allow-empty', '--no-gpg-sign', *args) + if all_files: # allow skipping `-a` with `all_files=False` + cmd += ('-a',) if msg is not None: # allow skipping `-m` with `msg=None` cmd += ('-m', msg) - return fn(*cmd, **kwargs) + ret, out, _ = fn(*cmd, **kwargs) + return ret, out.replace('\r\n', '\n') diff --git a/testing/zipapp/Dockerfile b/testing/zipapp/Dockerfile new file mode 100644 index 00000000..ea967e38 --- /dev/null +++ b/testing/zipapp/Dockerfile @@ -0,0 +1,14 @@ +FROM ubuntu:jammy +RUN : \ + && apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + python3 \ + python3-distutils \ + python3-venv \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +ENV LANG=C.UTF-8 PATH=/venv/bin:$PATH +RUN : \ + && python3 -mvenv /venv \ + && pip install --no-cache-dir pip distlib no-manylinux --upgrade diff --git a/testing/zipapp/entry b/testing/zipapp/entry new file mode 100755 index 00000000..15758d93 --- /dev/null +++ b/testing/zipapp/entry @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import os.path +import shutil +import stat +import sys +import tempfile +import zipfile + +from pre_commit.file_lock import lock + +CACHE_DIR = os.path.expanduser('~/.cache/pre-commit-zipapp') + + +def _make_executable(filename: str) -> None: + os.chmod(filename, os.stat(filename).st_mode | stat.S_IXUSR) + + +def _ensure_cache(zipf: zipfile.ZipFile, cache_key: str) -> str: + os.makedirs(CACHE_DIR, exist_ok=True) + + cache_dest = os.path.join(CACHE_DIR, cache_key) + lock_filename = os.path.join(CACHE_DIR, f'{cache_key}.lock') + + if os.path.exists(cache_dest): + return cache_dest + + with lock(lock_filename, blocked_cb=lambda: None): + # another process may have completed this work + if os.path.exists(cache_dest): + return cache_dest + + tmpdir = tempfile.mkdtemp(prefix=os.path.join(CACHE_DIR, '')) + try: + zipf.extractall(tmpdir) + # zip doesn't maintain permissions + _make_executable(os.path.join(tmpdir, 'python')) + _make_executable(os.path.join(tmpdir, 'python.exe')) + os.rename(tmpdir, cache_dest) + except BaseException: + shutil.rmtree(tmpdir) + raise + + return cache_dest + + +def main() -> int: + with zipfile.ZipFile(os.path.dirname(__file__)) as zipf: + with zipf.open('CACHE_KEY') as f: + cache_key = f.read().decode().strip() + + cache_dest = _ensure_cache(zipf, cache_key) + + if sys.platform != 'win32': + exe = os.path.join(cache_dest, 'python') + else: + exe = os.path.join(cache_dest, 'python.exe') + + cmd = (exe, '-mpre_commit', *sys.argv[1:]) + if sys.platform == 'win32': # https://bugs.python.org/issue19124 + import subprocess + + return subprocess.call(cmd) + else: + os.execvp(cmd[0], cmd) + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/testing/zipapp/make b/testing/zipapp/make new file mode 100755 index 00000000..43bb4373 --- /dev/null +++ b/testing/zipapp/make @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import hashlib +import io +import os.path +import shutil +import subprocess +import tempfile +import zipapp +import zipfile + +HERE = os.path.dirname(os.path.realpath(__file__)) +IMG = 'make-pre-commit-zipapp' + + +def _msg(s: str) -> None: + print(f'\033[7m{s}\033[m') + + +def _exit_if_retv(*cmd: str) -> None: + if subprocess.call(cmd): + raise SystemExit(1) + + +def _check_no_shared_objects(wheeldir: str) -> None: + for zip_filename in os.listdir(wheeldir): + with zipfile.ZipFile(os.path.join(wheeldir, zip_filename)) as zipf: + for filename in zipf.namelist(): + if filename.endswith('.so') or '.so.' in filename: + raise AssertionError(zip_filename, filename) + + +def _add_shim(dest: str) -> None: + shim = os.path.join(HERE, 'python') + shutil.copy(shim, dest) + + bio = io.BytesIO() + with zipfile.ZipFile(bio, 'w') as zipf: + zipf.write(shim, arcname='__main__.py') + + with tempfile.TemporaryDirectory() as tmpdir: + _exit_if_retv( + 'podman', 'run', '--rm', '--volume', f'{tmpdir}:/out:rw', IMG, + 'cp', '/venv/lib/python3.10/site-packages/distlib/t32.exe', '/out', + ) + + with open(os.path.join(dest, 'python.exe'), 'wb') as f: + with open(os.path.join(tmpdir, 't32.exe'), 'rb') as t32: + f.write(t32.read()) + f.write(b'#!py.exe -3\n') + f.write(bio.getvalue()) + + +def _write_cache_key(version: str, wheeldir: str, dest: str) -> None: + cache_hash = hashlib.sha256(f'{version}\n'.encode()) + for filename in sorted(os.listdir(wheeldir)): + cache_hash.update(f'{filename}\n'.encode()) + with open(os.path.join(HERE, 'python'), 'rb') as f: + cache_hash.update(f.read()) + with open(os.path.join(dest, 'CACHE_KEY'), 'wb') as f: + f.write(base64.urlsafe_b64encode(cache_hash.digest()).rstrip(b'=')) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument('version') + args = parser.parse_args() + + with tempfile.TemporaryDirectory() as tmpdir: + wheeldir = os.path.join(tmpdir, 'wheels') + os.mkdir(wheeldir) + + _msg('building podman image...') + _exit_if_retv('podman', 'build', '-q', '-t', IMG, HERE) + + _msg('populating wheels...') + _exit_if_retv( + 'podman', 'run', '--rm', '--volume', f'{wheeldir}:/wheels:rw', IMG, + 'pip', 'wheel', f'pre_commit=={args.version}', 'setuptools', + '--wheel-dir', '/wheels', + ) + + _msg('validating wheels...') + _check_no_shared_objects(wheeldir) + + _msg('adding __main__.py...') + mainfile = os.path.join(tmpdir, '__main__.py') + shutil.copy(os.path.join(HERE, 'entry'), mainfile) + + _msg('adding shim...') + _add_shim(tmpdir) + + _msg('copying file_lock.py...') + file_lock_py = os.path.join(HERE, '../../pre_commit/file_lock.py') + file_lock_py_dest = os.path.join(tmpdir, 'pre_commit/file_lock.py') + os.makedirs(os.path.dirname(file_lock_py_dest)) + shutil.copy(file_lock_py, file_lock_py_dest) + + _msg('writing CACHE_KEY...') + _write_cache_key(args.version, wheeldir, tmpdir) + + filename = f'pre-commit-{args.version}.pyz' + _msg(f'writing {filename}...') + shebang = '/usr/bin/env python3' + zipapp.create_archive(tmpdir, filename, interpreter=shebang) + + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/testing/zipapp/python b/testing/zipapp/python new file mode 100755 index 00000000..67910fca --- /dev/null +++ b/testing/zipapp/python @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""A shim executable to put dependencies on sys.path""" +from __future__ import annotations + +import argparse +import os.path +import runpy +import sys + +# an exe-zipapp will have a __file__ of shim.exe/__main__.py +EXE = __file__ if os.path.isfile(__file__) else os.path.dirname(__file__) +EXE = os.path.realpath(EXE) +HERE = os.path.dirname(EXE) +WHEELDIR = os.path.join(HERE, 'wheels') +SITE_DIRS = frozenset(('dist-packages', 'site-packages')) + + +def main() -> int: + parser = argparse.ArgumentParser(add_help=False) + parser.add_argument('-m') + args, rest = parser.parse_known_args() + + if args.m: + # try and remove site-packages from sys.path so our packages win + sys.path[:] = [ + p for p in sys.path + if os.path.split(p)[1] not in SITE_DIRS + ] + for wheel in sorted(os.listdir(WHEELDIR)): + sys.path.append(os.path.join(WHEELDIR, wheel)) + if args.m == 'pre_commit' or args.m.startswith('pre_commit.'): + sys.executable = EXE + sys.argv[1:] = rest + runpy.run_module(args.m, run_name='__main__', alter_sys=True) + return 0 + else: + cmd = (sys.executable, *sys.argv[1:]) + if sys.platform == 'win32': # https://bugs.python.org/issue19124 + import subprocess + + return subprocess.call(cmd) + else: + os.execvp(cmd[0], cmd) + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/tests/clientlib_test.py b/tests/clientlib_test.py index 6174889a..2c42b80c 100644 --- a/tests/clientlib_test.py +++ b/tests/clientlib_test.py @@ -1,6 +1,7 @@ -from __future__ import unicode_literals +from __future__ import annotations import logging +import re import cfgv import pytest @@ -11,10 +12,14 @@ from pre_commit.clientlib import CONFIG_HOOK_DICT from pre_commit.clientlib import CONFIG_REPO_DICT from pre_commit.clientlib import CONFIG_SCHEMA from pre_commit.clientlib import DEFAULT_LANGUAGE_VERSION +from pre_commit.clientlib import InvalidManifestError +from pre_commit.clientlib import load_manifest +from pre_commit.clientlib import MANIFEST_HOOK_DICT from pre_commit.clientlib import MANIFEST_SCHEMA -from pre_commit.clientlib import MigrateShaToRev -from pre_commit.clientlib import validate_config_main -from pre_commit.clientlib import validate_manifest_main +from pre_commit.clientlib import META_HOOK_DICT +from pre_commit.clientlib import OptionalSensibleRegexAtHook +from pre_commit.clientlib import OptionalSensibleRegexAtTop +from pre_commit.clientlib import parse_version from testing.fixtures import sample_local_config @@ -32,57 +37,56 @@ def test_check_type_tag_failures(value): check_type_tag(value) +def test_check_type_tag_success(): + check_type_tag('file') + + @pytest.mark.parametrize( - ('config_obj', 'expected'), ( - ( - { - 'repos': [{ - 'repo': 'git@github.com:pre-commit/pre-commit-hooks', - 'rev': 'cd74dc150c142c3be70b24eaf0b02cae9d235f37', - 'hooks': [{'id': 'pyflakes', 'files': '\\.py$'}], - }], - }, - True, - ), - ( - { - 'repos': [{ - 'repo': 'git@github.com:pre-commit/pre-commit-hooks', - 'rev': 'cd74dc150c142c3be70b24eaf0b02cae9d235f37', - 'hooks': [ - { - 'id': 'pyflakes', - 'files': '\\.py$', - 'args': ['foo', 'bar', 'baz'], - }, - ], - }], - }, - True, - ), - ( - { - 'repos': [{ - 'repo': 'git@github.com:pre-commit/pre-commit-hooks', - 'rev': 'cd74dc150c142c3be70b24eaf0b02cae9d235f37', - 'hooks': [ - { - 'id': 'pyflakes', - 'files': '\\.py$', - # Exclude pattern must be a string - 'exclude': 0, - 'args': ['foo', 'bar', 'baz'], - }, - ], - }], - }, - False, - ), + 'cfg', + ( + { + 'repos': [{ + 'repo': 'git@github.com:pre-commit/pre-commit-hooks', + 'rev': 'cd74dc150c142c3be70b24eaf0b02cae9d235f37', + 'hooks': [{'id': 'pyflakes', 'files': '\\.py$'}], + }], + }, + { + 'repos': [{ + 'repo': 'git@github.com:pre-commit/pre-commit-hooks', + 'rev': 'cd74dc150c142c3be70b24eaf0b02cae9d235f37', + 'hooks': [ + { + 'id': 'pyflakes', + 'files': '\\.py$', + 'args': ['foo', 'bar', 'baz'], + }, + ], + }], + }, ), ) -def test_config_valid(config_obj, expected): - ret = is_valid_according_to_schema(config_obj, CONFIG_SCHEMA) - assert ret is expected +def test_config_valid(cfg): + assert is_valid_according_to_schema(cfg, CONFIG_SCHEMA) + + +def test_invalid_config_wrong_type(): + cfg = { + 'repos': [{ + 'repo': 'git@github.com:pre-commit/pre-commit-hooks', + 'rev': 'cd74dc150c142c3be70b24eaf0b02cae9d235f37', + 'hooks': [ + { + 'id': 'pyflakes', + 'files': '\\.py$', + # Exclude pattern must be a string + 'exclude': 0, + 'args': ['foo', 'bar', 'baz'], + }, + ], + }], + } + assert not is_valid_according_to_schema(cfg, CONFIG_SCHEMA) def test_local_hooks_with_rev_fails(): @@ -104,158 +108,331 @@ def test_config_schema_does_not_contain_defaults(): assert not isinstance(item, cfgv.Optional) -def test_validate_manifest_main_ok(): - assert not validate_manifest_main(('.pre-commit-hooks.yaml',)) +def test_ci_map_key_allowed_at_top_level(caplog): + cfg = { + 'ci': {'skip': ['foo']}, + 'repos': [{'repo': 'meta', 'hooks': [{'id': 'identity'}]}], + } + cfgv.validate(cfg, CONFIG_SCHEMA) + assert not caplog.record_tuples -def test_validate_config_main_ok(): - assert not validate_config_main(('.pre-commit-config.yaml',)) - - -def test_validate_config_old_list_format_ok(tmpdir): - f = tmpdir.join('cfg.yaml') - f.write('- {repo: meta, hooks: [{id: identity}]}') - assert not validate_config_main((f.strpath,)) - - -def test_validate_warn_on_unknown_keys_at_repo_level(tmpdir, caplog): - f = tmpdir.join('cfg.yaml') - f.write( - '- repo: https://gitlab.com/pycqa/flake8\n' - ' rev: 3.7.7\n' - ' hooks:\n' - ' - id: flake8\n' - ' args: [--some-args]\n', - ) - ret_val = validate_config_main((f.strpath,)) - assert not ret_val - assert caplog.record_tuples == [ - ( - 'pre_commit', - logging.WARNING, - 'Unexpected key(s) present on https://gitlab.com/pycqa/flake8: ' - 'args', - ), - ] - - -def test_validate_warn_on_unknown_keys_at_top_level(tmpdir, caplog): - f = tmpdir.join('cfg.yaml') - f.write( - 'repos:\n' - '- repo: https://gitlab.com/pycqa/flake8\n' - ' rev: 3.7.7\n' - ' hooks:\n' - ' - id: flake8\n' - 'foo:\n' - ' id: 1.0.0\n', - ) - ret_val = validate_config_main((f.strpath,)) - assert not ret_val - assert caplog.record_tuples == [ - ( - 'pre_commit', - logging.WARNING, - 'Unexpected key(s) present at root: foo', - ), - ] - - -@pytest.mark.parametrize('fn', (validate_config_main, validate_manifest_main)) -def test_mains_not_ok(tmpdir, fn): - not_yaml = tmpdir.join('f.notyaml') - not_yaml.write('{') - not_schema = tmpdir.join('notconfig.yaml') - not_schema.write('{}') - - assert fn(('does-not-exist',)) - assert fn((not_yaml.strpath,)) - assert fn((not_schema.strpath,)) - - -@pytest.mark.parametrize( - ('manifest_obj', 'expected'), - ( - ( - [{ - 'id': 'a', - 'name': 'b', - 'entry': 'c', - 'language': 'python', - 'files': r'\.py$', - }], - True, - ), - ( - [{ - 'id': 'a', - 'name': 'b', - 'entry': 'c', - 'language': 'python', - 'language_version': 'python3.4', - 'files': r'\.py$', - }], - True, - ), - ( - # A regression in 0.13.5: always_run and files are permissible - [{ - 'id': 'a', - 'name': 'b', - 'entry': 'c', - 'language': 'python', - 'files': '', - 'always_run': True, - }], - True, - ), - ), -) -def test_valid_manifests(manifest_obj, expected): - ret = is_valid_according_to_schema(manifest_obj, MANIFEST_SCHEMA) - assert ret is expected - - -@pytest.mark.parametrize( - 'dct', - ( - {'repo': 'local'}, {'repo': 'meta'}, - {'repo': 'wat', 'sha': 'wat'}, {'repo': 'wat', 'rev': 'wat'}, - ), -) -def test_migrate_sha_to_rev_ok(dct): - MigrateShaToRev().check(dct) - - -def test_migrate_sha_to_rev_dont_specify_both(): - with pytest.raises(cfgv.ValidationError) as excinfo: - MigrateShaToRev().check({'repo': 'a', 'sha': 'b', 'rev': 'c'}) - msg, = excinfo.value.args - assert msg == 'Cannot specify both sha and rev' - - -@pytest.mark.parametrize( - 'dct', - ( - {'repo': 'a'}, - {'repo': 'meta', 'sha': 'a'}, {'repo': 'meta', 'rev': 'a'}, - ), -) -def test_migrate_sha_to_rev_conditional_check_failures(dct): +def test_ci_key_must_be_map(): with pytest.raises(cfgv.ValidationError): - MigrateShaToRev().check(dct) + cfgv.validate({'ci': 'invalid', 'repos': []}, CONFIG_SCHEMA) -def test_migrate_to_sha_apply_default(): - dct = {'repo': 'a', 'sha': 'b'} - MigrateShaToRev().apply_default(dct) - assert dct == {'repo': 'a', 'rev': 'b'} +@pytest.mark.parametrize( + 'rev', + ( + 'v0.12.4', + 'b27f281', + 'b27f281eb9398fc8504415d7fbdabf119ea8c5e1', + '19.10b0', + '4.3.21-2', + ), +) +def test_warn_mutable_rev_ok(caplog, rev): + config_obj = { + 'repo': 'https://gitlab.com/pycqa/flake8', + 'rev': rev, + 'hooks': [{'id': 'flake8'}], + } + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + assert caplog.record_tuples == [] -def test_migrate_to_sha_ok(): - dct = {'repo': 'a', 'rev': 'b'} - MigrateShaToRev().apply_default(dct) - assert dct == {'repo': 'a', 'rev': 'b'} +@pytest.mark.parametrize( + 'rev', + ( + '', + 'HEAD', + 'stable', + 'master', + 'some_branch_name', + ), +) +def test_warn_mutable_rev_invalid(caplog, rev): + config_obj = { + 'repo': 'https://gitlab.com/pycqa/flake8', + 'rev': rev, + 'hooks': [{'id': 'flake8'}], + } + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + "The 'rev' field of repo 'https://gitlab.com/pycqa/flake8' " + 'appears to be a mutable reference (moving tag / branch). ' + 'Mutable references are never updated after first install and are ' + 'not supported. ' + 'See https://pre-commit.com/#using-the-latest-version-for-a-repository ' # noqa: E501 + 'for more details. ' + 'Hint: `pre-commit autoupdate` often fixes this.', + ), + ] + + +def test_warn_mutable_rev_conditional(): + config_obj = { + 'repo': 'meta', + 'rev': '3.7.7', + 'hooks': [{'id': 'flake8'}], + } + + with pytest.raises(cfgv.ValidationError): + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + +@pytest.mark.parametrize( + 'validator_cls', + ( + OptionalSensibleRegexAtHook, + OptionalSensibleRegexAtTop, + ), +) +def test_sensible_regex_validators_dont_pass_none(validator_cls): + validator = validator_cls('files', cfgv.check_string) + with pytest.raises(cfgv.ValidationError) as excinfo: + validator.check({'files': None}) + + assert str(excinfo.value) == ( + '\n' + '==> At key: files' + '\n' + '=====> Expected string got NoneType' + ) + + +@pytest.mark.parametrize( + ('regex', 'warning'), + ( + ( + r'dir/*.py', + "The 'files' field in hook 'flake8' is a regex, not a glob -- " + "matching '/*' probably isn't what you want here", + ), + ( + r'dir[\/].*\.py', + r"pre-commit normalizes slashes in the 'files' field in hook " + r"'flake8' to forward slashes, so you can use / instead of [\/]", + ), + ( + r'dir[/\\].*\.py', + r"pre-commit normalizes slashes in the 'files' field in hook " + r"'flake8' to forward slashes, so you can use / instead of [/\\]", + ), + ( + r'dir[\\/].*\.py', + r"pre-commit normalizes slashes in the 'files' field in hook " + r"'flake8' to forward slashes, so you can use / instead of [\\/]", + ), + ), +) +def test_validate_optional_sensible_regex_at_hook(caplog, regex, warning): + config_obj = { + 'id': 'flake8', + 'files': regex, + } + cfgv.validate(config_obj, CONFIG_HOOK_DICT) + + assert caplog.record_tuples == [('pre_commit', logging.WARNING, warning)] + + +def test_validate_optional_sensible_regex_at_local_hook(caplog): + config_obj = sample_local_config() + config_obj['hooks'][0]['files'] = 'dir/*.py' + + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + "The 'files' field in hook 'do_not_commit' is a regex, not a glob " + "-- matching '/*' probably isn't what you want here", + ), + ] + + +def test_validate_optional_sensible_regex_at_meta_hook(caplog): + config_obj = { + 'repo': 'meta', + 'hooks': [{'id': 'identity', 'files': 'dir/*.py'}], + } + + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + "The 'files' field in hook 'identity' is a regex, not a glob " + "-- matching '/*' probably isn't what you want here", + ), + ] + + +@pytest.mark.parametrize( + ('regex', 'warning'), + ( + ( + r'dir/*.py', + "The top-level 'files' field is a regex, not a glob -- " + "matching '/*' probably isn't what you want here", + ), + ( + r'dir[\/].*\.py', + r"pre-commit normalizes the slashes in the top-level 'files' " + r'field to forward slashes, so you can use / instead of [\/]', + ), + ( + r'dir[/\\].*\.py', + r"pre-commit normalizes the slashes in the top-level 'files' " + r'field to forward slashes, so you can use / instead of [/\\]', + ), + ( + r'dir[\\/].*\.py', + r"pre-commit normalizes the slashes in the top-level 'files' " + r'field to forward slashes, so you can use / instead of [\\/]', + ), + ), +) +def test_validate_optional_sensible_regex_at_top_level(caplog, regex, warning): + config_obj = { + 'files': regex, + 'repos': [], + } + cfgv.validate(config_obj, CONFIG_SCHEMA) + + assert caplog.record_tuples == [('pre_commit', logging.WARNING, warning)] + + +def test_invalid_stages_error(): + cfg = {'repos': [sample_local_config()]} + cfg['repos'][0]['hooks'][0]['stages'] = ['invalid'] + + with pytest.raises(cfgv.ValidationError) as excinfo: + cfgv.validate(cfg, CONFIG_SCHEMA) + + assert str(excinfo.value) == ( + '\n' + '==> At Config()\n' + '==> At key: repos\n' + "==> At Repository(repo='local')\n" + '==> At key: hooks\n' + "==> At Hook(id='do_not_commit')\n" + # this line was missing due to the custom validator + '==> At key: stages\n' + '==> At index 0\n' + "=====> Expected one of commit-msg, manual, post-checkout, post-commit, post-merge, post-rewrite, pre-commit, pre-merge-commit, pre-push, pre-rebase, prepare-commit-msg but got: 'invalid'" # noqa: E501 + ) + + +def test_warning_for_deprecated_stages(caplog): + config_obj = sample_local_config() + config_obj['hooks'][0]['stages'] = ['commit', 'push'] + + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + 'hook id `do_not_commit` uses deprecated stage names ' + '(commit, push) which will be removed in a future version. ' + 'run: `pre-commit migrate-config` to automatically fix this.', + ), + ] + + +def test_no_warning_for_non_deprecated_stages(caplog): + config_obj = sample_local_config() + config_obj['hooks'][0]['stages'] = ['pre-commit', 'pre-push'] + + cfgv.validate(config_obj, CONFIG_REPO_DICT) + + assert caplog.record_tuples == [] + + +def test_warning_for_deprecated_default_stages(caplog): + cfg = {'default_stages': ['commit', 'push'], 'repos': []} + + cfgv.validate(cfg, CONFIG_SCHEMA) + + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + 'top-level `default_stages` uses deprecated stage names ' + '(commit, push) which will be removed in a future version. ' + 'run: `pre-commit migrate-config` to automatically fix this.', + ), + ] + + +def test_no_warning_for_non_deprecated_default_stages(caplog): + cfg = {'default_stages': ['pre-commit', 'pre-push'], 'repos': []} + + cfgv.validate(cfg, CONFIG_SCHEMA) + + assert caplog.record_tuples == [] + + +def test_unsupported_language_migration(): + cfg = {'repos': [sample_local_config(), sample_local_config()]} + cfg['repos'][0]['hooks'][0]['language'] = 'system' + cfg['repos'][1]['hooks'][0]['language'] = 'script' + + cfgv.validate(cfg, CONFIG_SCHEMA) + ret = cfgv.apply_defaults(cfg, CONFIG_SCHEMA) + + assert ret['repos'][0]['hooks'][0]['language'] == 'unsupported' + assert ret['repos'][1]['hooks'][0]['language'] == 'unsupported_script' + + +def test_unsupported_language_migration_language_required(): + cfg = {'repos': [sample_local_config()]} + del cfg['repos'][0]['hooks'][0]['language'] + + with pytest.raises(cfgv.ValidationError): + cfgv.validate(cfg, CONFIG_SCHEMA) + + +@pytest.mark.parametrize( + 'manifest_obj', + ( + [{ + 'id': 'a', + 'name': 'b', + 'entry': 'c', + 'language': 'python', + 'files': r'\.py$', + }], + [{ + 'id': 'a', + 'name': 'b', + 'entry': 'c', + 'language': 'python', + 'language_version': 'python3.4', + 'files': r'\.py$', + }], + # A regression in 0.13.5: always_run and files are permissible + [{ + 'id': 'a', + 'name': 'b', + 'entry': 'c', + 'language': 'python', + 'files': '', + 'always_run': True, + }], + ), +) +def test_valid_manifests(manifest_obj): + assert is_valid_according_to_schema(manifest_obj, MANIFEST_SCHEMA) @pytest.mark.parametrize( @@ -267,6 +444,13 @@ def test_migrate_to_sha_ok(): {'repo': 'meta', 'hooks': [{'id': 'identity', 'language': 'python'}]}, # name override must be string {'repo': 'meta', 'hooks': [{'id': 'identity', 'name': False}]}, + pytest.param( + { + 'repo': 'meta', + 'hooks': [{'id': 'identity', 'entry': 'echo hi'}], + }, + id='cannot override entry for meta hooks', + ), ), ) def test_meta_hook_invalid(config_repo): @@ -274,6 +458,15 @@ def test_meta_hook_invalid(config_repo): cfgv.validate(config_repo, CONFIG_REPO_DICT) +def test_meta_check_hooks_apply_only_at_top_level(): + cfg = {'id': 'check-hooks-apply'} + cfg = cfgv.apply_defaults(cfg, META_HOOK_DICT) + + files_re = re.compile(cfg['files']) + assert files_re.search('.pre-commit-config.yaml') + assert not files_re.search('foo/.pre-commit-config.yaml') + + @pytest.mark.parametrize( 'mapping', ( @@ -288,18 +481,53 @@ def test_default_language_version_invalid(mapping): cfgv.validate(mapping, DEFAULT_LANGUAGE_VERSION) +def test_parse_version(): + assert parse_version('0.0') == parse_version('0.0') + assert parse_version('0.1') > parse_version('0.0') + assert parse_version('2.1') >= parse_version('2') + + def test_minimum_pre_commit_version_failing(): + cfg = {'repos': [], 'minimum_pre_commit_version': '999'} with pytest.raises(cfgv.ValidationError) as excinfo: - cfg = {'repos': [], 'minimum_pre_commit_version': '999'} cfgv.validate(cfg, CONFIG_SCHEMA) assert str(excinfo.value) == ( - '\n' - '==> At Config()\n' - '==> At key: minimum_pre_commit_version\n' - '=====> pre-commit version 999 is required but version {} is ' - 'installed. Perhaps run `pip install --upgrade pre-commit`.'.format( - C.VERSION, - ) + f'\n' + f'==> At Config()\n' + f'==> At key: minimum_pre_commit_version\n' + f'=====> pre-commit version 999 is required but version {C.VERSION} ' + f'is installed. Perhaps run `pip install --upgrade pre-commit`.' + ) + + +def test_minimum_pre_commit_version_failing_in_config(): + cfg = {'repos': [sample_local_config()]} + cfg['repos'][0]['hooks'][0]['minimum_pre_commit_version'] = '999' + with pytest.raises(cfgv.ValidationError) as excinfo: + cfgv.validate(cfg, CONFIG_SCHEMA) + assert str(excinfo.value) == ( + f'\n' + f'==> At Config()\n' + f'==> At key: repos\n' + f"==> At Repository(repo='local')\n" + f'==> At key: hooks\n' + f"==> At Hook(id='do_not_commit')\n" + f'==> At key: minimum_pre_commit_version\n' + f'=====> pre-commit version 999 is required but version {C.VERSION} ' + f'is installed. Perhaps run `pip install --upgrade pre-commit`.' + ) + + +def test_minimum_pre_commit_version_failing_before_other_error(): + cfg = {'repos': 5, 'minimum_pre_commit_version': '999'} + with pytest.raises(cfgv.ValidationError) as excinfo: + cfgv.validate(cfg, CONFIG_SCHEMA) + assert str(excinfo.value) == ( + f'\n' + f'==> At Config()\n' + f'==> At key: minimum_pre_commit_version\n' + f'=====> pre-commit version 999 is required but version {C.VERSION} ' + f'is installed. Perhaps run `pip install --upgrade pre-commit`.' ) @@ -311,7 +539,69 @@ def test_minimum_pre_commit_version_passing(): @pytest.mark.parametrize('schema', (CONFIG_SCHEMA, CONFIG_REPO_DICT)) def test_warn_additional(schema): allowed_keys = {item.key for item in schema.items if hasattr(item, 'key')} - warn_additional, = [ + warn_additional, = ( x for x in schema.items if isinstance(x, cfgv.WarnAdditionalKeys) - ] + ) assert allowed_keys == set(warn_additional.keys) + + +def test_stages_migration_for_default_stages(): + cfg = { + 'default_stages': ['commit-msg', 'push', 'commit', 'merge-commit'], + 'repos': [], + } + cfgv.validate(cfg, CONFIG_SCHEMA) + cfg = cfgv.apply_defaults(cfg, CONFIG_SCHEMA) + assert cfg['default_stages'] == [ + 'commit-msg', 'pre-push', 'pre-commit', 'pre-merge-commit', + ] + + +def test_manifest_stages_defaulting(): + dct = { + 'id': 'fake-hook', + 'name': 'fake-hook', + 'entry': 'fake-hook', + 'language': 'system', + 'stages': ['commit-msg', 'push', 'commit', 'merge-commit'], + } + cfgv.validate(dct, MANIFEST_HOOK_DICT) + dct = cfgv.apply_defaults(dct, MANIFEST_HOOK_DICT) + assert dct['stages'] == [ + 'commit-msg', 'pre-push', 'pre-commit', 'pre-merge-commit', + ] + + +def test_config_hook_stages_defaulting_missing(): + dct = {'id': 'fake-hook'} + cfgv.validate(dct, CONFIG_HOOK_DICT) + dct = cfgv.apply_defaults(dct, CONFIG_HOOK_DICT) + assert dct == {'id': 'fake-hook'} + + +def test_config_hook_stages_defaulting(): + dct = { + 'id': 'fake-hook', + 'stages': ['commit-msg', 'push', 'commit', 'merge-commit'], + } + cfgv.validate(dct, CONFIG_HOOK_DICT) + dct = cfgv.apply_defaults(dct, CONFIG_HOOK_DICT) + assert dct == { + 'id': 'fake-hook', + 'stages': ['commit-msg', 'pre-push', 'pre-commit', 'pre-merge-commit'], + } + + +def test_manifest_v5_forward_compat(tmp_path): + manifest = tmp_path.joinpath('.pre-commit-hooks.yaml') + manifest.write_text('hooks: {}') + + with pytest.raises(InvalidManifestError) as excinfo: + load_manifest(manifest) + assert str(excinfo.value) == ( + f'\n' + f'==> File {manifest}\n' + f'=====> \n' + f'=====> pre-commit version 5 is required but version {C.VERSION} ' + f'is installed. Perhaps run `pip install --upgrade pre-commit`.' + ) diff --git a/tests/color_test.py b/tests/color_test.py index 6c9889d1..89b4fd3e 100644 --- a/tests/color_test.py +++ b/tests/color_test.py @@ -1,20 +1,19 @@ -from __future__ import unicode_literals +from __future__ import annotations import sys +from unittest import mock -import mock import pytest from pre_commit import envcontext from pre_commit.color import format_color from pre_commit.color import GREEN -from pre_commit.color import InvalidColorSetting from pre_commit.color import use_color @pytest.mark.parametrize( ('in_text', 'in_color', 'in_use_color', 'expected'), ( - ('foo', GREEN, True, '{}foo\033[0m'.format(GREEN)), + ('foo', GREEN, True, f'{GREEN}foo\033[m'), ('foo', GREEN, False, 'foo'), ), ) @@ -32,31 +31,31 @@ def test_use_color_always(): def test_use_color_no_tty(): - with mock.patch.object(sys.stdout, 'isatty', return_value=False): + with mock.patch.object(sys.stderr, 'isatty', return_value=False): assert use_color('auto') is False def test_use_color_tty_with_color_support(): - with mock.patch.object(sys.stdout, 'isatty', return_value=True): + with mock.patch.object(sys.stderr, 'isatty', return_value=True): with mock.patch('pre_commit.color.terminal_supports_color', True): - with envcontext.envcontext([('TERM', envcontext.UNSET)]): + with envcontext.envcontext((('TERM', envcontext.UNSET),)): assert use_color('auto') is True def test_use_color_tty_without_color_support(): - with mock.patch.object(sys.stdout, 'isatty', return_value=True): + with mock.patch.object(sys.stderr, 'isatty', return_value=True): with mock.patch('pre_commit.color.terminal_supports_color', False): - with envcontext.envcontext([('TERM', envcontext.UNSET)]): + with envcontext.envcontext((('TERM', envcontext.UNSET),)): assert use_color('auto') is False def test_use_color_dumb_term(): - with mock.patch.object(sys.stdout, 'isatty', return_value=True): + with mock.patch.object(sys.stderr, 'isatty', return_value=True): with mock.patch('pre_commit.color.terminal_supports_color', True): - with envcontext.envcontext([('TERM', 'dumb')]): + with envcontext.envcontext((('TERM', 'dumb'),)): assert use_color('auto') is False def test_use_color_raises_if_given_shenanigans(): - with pytest.raises(InvalidColorSetting): + with pytest.raises(ValueError): use_color('herpaderp') diff --git a/tests/commands/autoupdate_test.py b/tests/commands/autoupdate_test.py index ead0efe5..71bd0444 100644 --- a/tests/commands/autoupdate_test.py +++ b/tests/commands/autoupdate_test.py @@ -1,14 +1,18 @@ -from __future__ import unicode_literals +from __future__ import annotations -import pipes +import shlex +from unittest import mock import pytest import pre_commit.constants as C +from pre_commit import envcontext from pre_commit import git -from pre_commit.commands.autoupdate import _update_repo +from pre_commit import yaml +from pre_commit.commands.autoupdate import _check_hooks_still_exist_at_rev from pre_commit.commands.autoupdate import autoupdate from pre_commit.commands.autoupdate import RepositoryCannotBeUpdatedError +from pre_commit.commands.autoupdate import RevInfo from pre_commit.util import cmd_output from testing.auto_namedtuple import auto_namedtuple from testing.fixtures import add_config_to_repo @@ -22,33 +26,135 @@ from testing.util import git_commit @pytest.fixture -def up_to_date_repo(tempdir_factory): +def up_to_date(tempdir_factory): yield make_repo(tempdir_factory, 'python_hooks_repo') -def test_up_to_date_repo(up_to_date_repo, store): - config = make_config_from_repo(up_to_date_repo) - input_rev = config['rev'] - ret = _update_repo(config, store, tags_only=False) - assert ret['rev'] == input_rev +@pytest.fixture +def out_of_date(tempdir_factory): + path = make_repo(tempdir_factory, 'python_hooks_repo') + original_rev = git.head_rev(path) + + git_commit(cwd=path) + head_rev = git.head_rev(path) + + yield auto_namedtuple( + path=path, original_rev=original_rev, head_rev=head_rev, + ) -def test_autoupdate_up_to_date_repo(up_to_date_repo, in_tmpdir, store): - # Write out the config - config = make_config_from_repo(up_to_date_repo, check=False) - write_config('.', config) - - with open(C.CONFIG_FILE) as f: - before = f.read() - assert '^$' not in before - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False) - with open(C.CONFIG_FILE) as f: - after = f.read() - assert ret == 0 - assert before == after +@pytest.fixture +def tagged(out_of_date): + cmd_output('git', 'tag', 'v1.2.3', cwd=out_of_date.path) + yield out_of_date -def test_autoupdate_old_revision_broken(tempdir_factory, in_tmpdir, store): +@pytest.fixture +def hook_disappearing(tempdir_factory): + path = make_repo(tempdir_factory, 'python_hooks_repo') + original_rev = git.head_rev(path) + + with modify_manifest(path) as manifest: + manifest[0]['id'] = 'bar' + + yield auto_namedtuple(path=path, original_rev=original_rev) + + +def test_rev_info_from_config(): + info = RevInfo.from_config({'repo': 'repo/path', 'rev': 'v1.2.3'}) + assert info == RevInfo('repo/path', 'v1.2.3', None) + + +def test_rev_info_update_up_to_date_repo(up_to_date): + config = make_config_from_repo(up_to_date) + info = RevInfo.from_config(config)._replace(hook_ids=frozenset(('foo',))) + new_info = info.update(tags_only=False, freeze=False) + assert info == new_info + + +def test_rev_info_update_out_of_date_repo(out_of_date): + config = make_config_from_repo( + out_of_date.path, rev=out_of_date.original_rev, + ) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=False, freeze=False) + assert new_info.rev == out_of_date.head_rev + + +def test_rev_info_update_non_master_default_branch(out_of_date): + # change the default branch to be not-master + cmd_output('git', '-C', out_of_date.path, 'branch', '-m', 'dev') + test_rev_info_update_out_of_date_repo(out_of_date) + + +def test_rev_info_update_tags_even_if_not_tags_only(tagged): + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=False, freeze=False) + assert new_info.rev == 'v1.2.3' + + +def test_rev_info_update_tags_only_does_not_pick_tip(tagged): + git_commit(cwd=tagged.path) + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=True, freeze=False) + assert new_info.rev == 'v1.2.3' + + +def test_rev_info_update_tags_prefers_version_tag(tagged, out_of_date): + cmd_output('git', 'tag', 'latest', cwd=out_of_date.path) + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=True, freeze=False) + assert new_info.rev == 'v1.2.3' + + +def test_rev_info_update_tags_non_version_tag(out_of_date): + cmd_output('git', 'tag', 'latest', cwd=out_of_date.path) + config = make_config_from_repo( + out_of_date.path, rev=out_of_date.original_rev, + ) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=True, freeze=False) + assert new_info.rev == 'latest' + + +def test_rev_info_update_freeze_tag(tagged): + git_commit(cwd=tagged.path) + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=True, freeze=True) + assert new_info.rev == tagged.head_rev + assert new_info.frozen == 'v1.2.3' + + +def test_rev_info_update_does_not_freeze_if_already_sha(out_of_date): + config = make_config_from_repo( + out_of_date.path, rev=out_of_date.original_rev, + ) + info = RevInfo.from_config(config) + new_info = info.update(tags_only=True, freeze=True) + assert new_info.rev == out_of_date.head_rev + assert new_info.frozen is None + + +def test_autoupdate_up_to_date_repo(up_to_date, tmpdir): + contents = ( + f'repos:\n' + f'- repo: {up_to_date}\n' + f' rev: {git.head_rev(up_to_date)}\n' + f' hooks:\n' + f' - id: foo\n' + ) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write(contents) + + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + assert cfg.read() == contents + + +def test_autoupdate_old_revision_broken(tempdir_factory, in_tmpdir): """In $FUTURE_VERSION, hooks.yaml will no longer be supported. This asserts that when that day comes, pre-commit will be able to autoupdate despite not being able to read hooks.yaml in that repository. @@ -68,98 +174,114 @@ def test_autoupdate_old_revision_broken(tempdir_factory, in_tmpdir, store): write_config('.', config) with open(C.CONFIG_FILE) as f: before = f.read() - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False) + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) == 0 with open(C.CONFIG_FILE) as f: after = f.read() - assert ret == 0 assert before != after assert update_rev in after -@pytest.fixture -def out_of_date_repo(tempdir_factory): - path = make_repo(tempdir_factory, 'python_hooks_repo') - original_rev = git.head_rev(path) - - git_commit(cwd=path) - head_rev = git.head_rev(path) - - yield auto_namedtuple( - path=path, original_rev=original_rev, head_rev=head_rev, +def test_autoupdate_out_of_date_repo(out_of_date, tmpdir): + fmt = ( + 'repos:\n' + '- repo: {}\n' + ' rev: {}\n' + ' hooks:\n' + ' - id: foo\n' ) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write(fmt.format(out_of_date.path, out_of_date.original_rev)) + + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + assert cfg.read() == fmt.format(out_of_date.path, out_of_date.head_rev) -def test_out_of_date_repo(out_of_date_repo, store): - config = make_config_from_repo( - out_of_date_repo.path, rev=out_of_date_repo.original_rev, +def test_autoupdate_with_core_useBuiltinFSMonitor(out_of_date, tmpdir): + # force the setting on "globally" for git + home = tmpdir.join('fakehome').ensure_dir() + home.join('.gitconfig').write('[core]\nuseBuiltinFSMonitor = true\n') + with envcontext.envcontext((('HOME', str(home)),)): + test_autoupdate_out_of_date_repo(out_of_date, tmpdir) + + +def test_autoupdate_pure_yaml(out_of_date, tmpdir): + with mock.patch.object(yaml, 'Dumper', yaml.yaml.SafeDumper): + test_autoupdate_out_of_date_repo(out_of_date, tmpdir) + + +def test_autoupdate_only_one_to_update(up_to_date, out_of_date, tmpdir): + fmt = ( + 'repos:\n' + '- repo: {}\n' + ' rev: {}\n' + ' hooks:\n' + ' - id: foo\n' + '- repo: {}\n' + ' rev: {}\n' + ' hooks:\n' + ' - id: foo\n' ) - ret = _update_repo(config, store, tags_only=False) - assert ret['rev'] != out_of_date_repo.original_rev - assert ret['rev'] == out_of_date_repo.head_rev - - -def test_autoupdate_out_of_date_repo(out_of_date_repo, in_tmpdir, store): - # Write out the config - config = make_config_from_repo( - out_of_date_repo.path, rev=out_of_date_repo.original_rev, check=False, + cfg = tmpdir.join(C.CONFIG_FILE) + before = fmt.format( + up_to_date, git.head_rev(up_to_date), + out_of_date.path, out_of_date.original_rev, ) - write_config('.', config) + cfg.write(before) - with open(C.CONFIG_FILE) as f: - before = f.read() - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False) - with open(C.CONFIG_FILE) as f: - after = f.read() - assert ret == 0 - assert before != after - # Make sure we don't add defaults - assert 'exclude' not in after - assert out_of_date_repo.head_rev in after + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + assert cfg.read() == fmt.format( + up_to_date, git.head_rev(up_to_date), + out_of_date.path, out_of_date.head_rev, + ) def test_autoupdate_out_of_date_repo_with_correct_repo_name( - out_of_date_repo, in_tmpdir, store, + out_of_date, in_tmpdir, ): stale_config = make_config_from_repo( - out_of_date_repo.path, rev=out_of_date_repo.original_rev, check=False, + out_of_date.path, rev=out_of_date.original_rev, check=False, ) local_config = sample_local_config() config = {'repos': [stale_config, local_config]} - # Write out the config write_config('.', config) with open(C.CONFIG_FILE) as f: before = f.read() - repo_name = 'file://{}'.format(out_of_date_repo.path) - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False, repos=(repo_name,)) + repo_name = f'file://{out_of_date.path}' + ret = autoupdate( + C.CONFIG_FILE, freeze=False, tags_only=False, + repos=(repo_name,), + ) with open(C.CONFIG_FILE) as f: after = f.read() assert ret == 0 assert before != after - assert out_of_date_repo.head_rev in after + assert out_of_date.head_rev in after assert 'local' in after def test_autoupdate_out_of_date_repo_with_wrong_repo_name( - out_of_date_repo, in_tmpdir, store, + out_of_date, in_tmpdir, ): - # Write out the config config = make_config_from_repo( - out_of_date_repo.path, rev=out_of_date_repo.original_rev, check=False, + out_of_date.path, rev=out_of_date.original_rev, check=False, ) write_config('.', config) with open(C.CONFIG_FILE) as f: before = f.read() # It will not update it, because the name doesn't match - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False, repos=('dne',)) + ret = autoupdate( + C.CONFIG_FILE, freeze=False, tags_only=False, + repos=('dne',), + ) with open(C.CONFIG_FILE) as f: after = f.read() assert ret == 0 assert before == after -def test_does_not_reformat(in_tmpdir, out_of_date_repo, store): +def test_does_not_reformat(tmpdir, out_of_date): fmt = ( 'repos:\n' '- repo: {}\n' @@ -169,20 +291,54 @@ def test_does_not_reformat(in_tmpdir, out_of_date_repo, store): ' # These args are because reasons!\n' ' args: [foo, bar, baz]\n' ) - config = fmt.format(out_of_date_repo.path, out_of_date_repo.original_rev) - with open(C.CONFIG_FILE, 'w') as f: - f.write(config) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write(fmt.format(out_of_date.path, out_of_date.original_rev)) - autoupdate(C.CONFIG_FILE, store, tags_only=False) - with open(C.CONFIG_FILE) as f: - after = f.read() - expected = fmt.format(out_of_date_repo.path, out_of_date_repo.head_rev) - assert after == expected + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + expected = fmt.format(out_of_date.path, out_of_date.head_rev) + assert cfg.read() == expected -def test_loses_formatting_when_not_detectable( - out_of_date_repo, store, in_tmpdir, -): +def test_does_not_change_mixed_endlines_read(up_to_date, tmpdir): + fmt = ( + 'repos:\n' + '- repo: {}\n' + ' rev: {} # definitely the version I want!\r\n' + ' hooks:\r\n' + ' - id: foo\n' + ' # These args are because reasons!\r\n' + ' args: [foo, bar, baz]\r\n' + ) + cfg = tmpdir.join(C.CONFIG_FILE) + + expected = fmt.format(up_to_date, git.head_rev(up_to_date)).encode() + cfg.write_binary(expected) + + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + assert cfg.read_binary() == expected + + +def test_does_not_change_mixed_endlines_write(tmpdir, out_of_date): + fmt = ( + 'repos:\n' + '- repo: {}\n' + ' rev: {} # definitely the version I want!\r\n' + ' hooks:\r\n' + ' - id: foo\n' + ' # These args are because reasons!\r\n' + ' args: [foo, bar, baz]\r\n' + ) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write_binary( + fmt.format(out_of_date.path, out_of_date.original_rev).encode(), + ) + + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + expected = fmt.format(out_of_date.path, out_of_date.head_rev).encode() + assert cfg.read_binary() == expected + + +def test_loses_formatting_when_not_detectable(out_of_date, tmpdir): """A best-effort attempt is made at updating rev without rewriting formatting. When the original formatting cannot be detected, this is abandoned. @@ -197,149 +353,124 @@ def test_loses_formatting_when_not_detectable( ' ],\n' ' }}\n' ']\n'.format( - pipes.quote(out_of_date_repo.path), out_of_date_repo.original_rev, + shlex.quote(out_of_date.path), out_of_date.original_rev, ) ) - with open(C.CONFIG_FILE, 'w') as f: - f.write(config) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write(config) - autoupdate(C.CONFIG_FILE, store, tags_only=False) - with open(C.CONFIG_FILE) as f: - after = f.read() + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 expected = ( - 'repos:\n' - '- repo: {}\n' - ' rev: {}\n' - ' hooks:\n' - ' - id: foo\n' - ).format(out_of_date_repo.path, out_of_date_repo.head_rev) - assert after == expected - - -@pytest.fixture -def tagged_repo(out_of_date_repo): - cmd_output('git', 'tag', 'v1.2.3', cwd=out_of_date_repo.path) - yield out_of_date_repo - - -def test_autoupdate_tagged_repo(tagged_repo, in_tmpdir, store): - config = make_config_from_repo( - tagged_repo.path, rev=tagged_repo.original_rev, + f'repos:\n' + f'- repo: {out_of_date.path}\n' + f' rev: {out_of_date.head_rev}\n' + f' hooks:\n' + f' - id: foo\n' ) + assert cfg.read() == expected + + +def test_autoupdate_tagged_repo(tagged, in_tmpdir): + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) write_config('.', config) - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False) - assert ret == 0 + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) == 0 with open(C.CONFIG_FILE) as f: assert 'v1.2.3' in f.read() -@pytest.fixture -def tagged_repo_with_more_commits(tagged_repo): - git_commit(cwd=tagged_repo.path) - yield tagged_repo - - -def test_autoupdate_tags_only(tagged_repo_with_more_commits, in_tmpdir, store): - config = make_config_from_repo( - tagged_repo_with_more_commits.path, - rev=tagged_repo_with_more_commits.original_rev, - ) +def test_autoupdate_freeze(tagged, in_tmpdir): + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) write_config('.', config) - ret = autoupdate(C.CONFIG_FILE, store, tags_only=True) - assert ret == 0 + assert autoupdate(C.CONFIG_FILE, freeze=True, tags_only=False) == 0 + with open(C.CONFIG_FILE) as f: + expected = f'rev: {tagged.head_rev} # frozen: v1.2.3' + assert expected in f.read() + + # if we un-freeze it should remove the frozen comment + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) == 0 + with open(C.CONFIG_FILE) as f: + assert 'rev: v1.2.3\n' in f.read() + + +def test_autoupdate_tags_only(tagged, in_tmpdir): + # add some commits after the tag + git_commit(cwd=tagged.path) + + config = make_config_from_repo(tagged.path, rev=tagged.original_rev) + write_config('.', config) + + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=True) == 0 with open(C.CONFIG_FILE) as f: assert 'v1.2.3' in f.read() -def test_autoupdate_latest_no_config(out_of_date_repo, in_tmpdir, store): +def test_autoupdate_latest_no_config(out_of_date, in_tmpdir): config = make_config_from_repo( - out_of_date_repo.path, rev=out_of_date_repo.original_rev, + out_of_date.path, rev=out_of_date.original_rev, ) write_config('.', config) - cmd_output('git', 'rm', '-r', ':/', cwd=out_of_date_repo.path) - git_commit(cwd=out_of_date_repo.path) + cmd_output('git', 'rm', '-r', ':/', cwd=out_of_date.path) + git_commit(cwd=out_of_date.path) - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False) - assert ret == 1 + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) == 1 with open(C.CONFIG_FILE) as f: - assert out_of_date_repo.original_rev in f.read() + assert out_of_date.original_rev in f.read() -@pytest.fixture -def hook_disappearing_repo(tempdir_factory): - path = make_repo(tempdir_factory, 'python_hooks_repo') - original_rev = git.head_rev(path) - - with modify_manifest(path) as manifest: - manifest[0]['id'] = 'bar' - - yield auto_namedtuple(path=path, original_rev=original_rev) - - -def test_hook_disppearing_repo_raises(hook_disappearing_repo, store): +def test_hook_disppearing_repo_raises(hook_disappearing): config = make_config_from_repo( - hook_disappearing_repo.path, - rev=hook_disappearing_repo.original_rev, + hook_disappearing.path, + rev=hook_disappearing.original_rev, hooks=[{'id': 'foo'}], ) + info = RevInfo.from_config(config).update(tags_only=False, freeze=False) with pytest.raises(RepositoryCannotBeUpdatedError): - _update_repo(config, store, tags_only=False) + _check_hooks_still_exist_at_rev(config, info) -def test_autoupdate_hook_disappearing_repo( - hook_disappearing_repo, in_tmpdir, store, -): - config = make_config_from_repo( - hook_disappearing_repo.path, - rev=hook_disappearing_repo.original_rev, - hooks=[{'id': 'foo'}], - check=False, +def test_autoupdate_hook_disappearing_repo(hook_disappearing, tmpdir): + contents = ( + f'repos:\n' + f'- repo: {hook_disappearing.path}\n' + f' rev: {hook_disappearing.original_rev}\n' + f' hooks:\n' + f' - id: foo\n' ) - write_config('.', config) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write(contents) - with open(C.CONFIG_FILE) as f: - before = f.read() - ret = autoupdate(C.CONFIG_FILE, store, tags_only=False) - with open(C.CONFIG_FILE) as f: - after = f.read() - assert ret == 1 - assert before == after + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 1 + assert cfg.read() == contents -def test_autoupdate_non_master_default_branch(up_to_date_repo, store): - # change the default branch to be not-master - cmd_output('git', '-C', up_to_date_repo, 'branch', '-m', 'dev') - test_up_to_date_repo(up_to_date_repo, store) - - -def test_autoupdate_local_hooks(in_git_dir, store): +def test_autoupdate_local_hooks(in_git_dir): config = sample_local_config() add_config_to_repo('.', config) - assert autoupdate(C.CONFIG_FILE, store, tags_only=False) == 0 - new_config_writen = read_config('.') - assert len(new_config_writen['repos']) == 1 - assert new_config_writen['repos'][0] == config + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) == 0 + new_config_written = read_config('.') + assert len(new_config_written['repos']) == 1 + assert new_config_written['repos'][0] == config def test_autoupdate_local_hooks_with_out_of_date_repo( - out_of_date_repo, in_tmpdir, store, + out_of_date, in_tmpdir, ): stale_config = make_config_from_repo( - out_of_date_repo.path, rev=out_of_date_repo.original_rev, check=False, + out_of_date.path, rev=out_of_date.original_rev, check=False, ) local_config = sample_local_config() config = {'repos': [local_config, stale_config]} write_config('.', config) - assert autoupdate(C.CONFIG_FILE, store, tags_only=False) == 0 - new_config_writen = read_config('.') - assert len(new_config_writen['repos']) == 2 - assert new_config_writen['repos'][0] == local_config + assert autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) == 0 + new_config_written = read_config('.') + assert len(new_config_written['repos']) == 2 + assert new_config_written['repos'][0] == local_config -def test_autoupdate_meta_hooks(tmpdir, capsys, store): +def test_autoupdate_meta_hooks(tmpdir): cfg = tmpdir.join(C.CONFIG_FILE) cfg.write( 'repos:\n' @@ -347,9 +478,7 @@ def test_autoupdate_meta_hooks(tmpdir, capsys, store): ' hooks:\n' ' - id: check-useless-excludes\n', ) - with tmpdir.as_cwd(): - ret = autoupdate(C.CONFIG_FILE, store, tags_only=True) - assert ret == 0 + assert autoupdate(str(cfg), freeze=False, tags_only=True) == 0 assert cfg.read() == ( 'repos:\n' '- repo: meta\n' @@ -358,7 +487,7 @@ def test_autoupdate_meta_hooks(tmpdir, capsys, store): ) -def test_updates_old_format_to_new_format(tmpdir, capsys, store): +def test_updates_old_format_to_new_format(tmpdir, capsys): cfg = tmpdir.join(C.CONFIG_FILE) cfg.write( '- repo: local\n' @@ -368,9 +497,7 @@ def test_updates_old_format_to_new_format(tmpdir, capsys, store): ' entry: ./bin/foo.sh\n' ' language: script\n', ) - with tmpdir.as_cwd(): - ret = autoupdate(C.CONFIG_FILE, store, tags_only=True) - assert ret == 0 + assert autoupdate(str(cfg), freeze=False, tags_only=True) == 0 contents = cfg.read() assert contents == ( 'repos:\n' @@ -383,3 +510,23 @@ def test_updates_old_format_to_new_format(tmpdir, capsys, store): ) out, _ = capsys.readouterr() assert out == 'Configuration has been migrated.\n' + + +def test_maintains_rev_quoting_style(tmpdir, out_of_date): + fmt = ( + 'repos:\n' + '- repo: {path}\n' + ' rev: "{rev}"\n' + ' hooks:\n' + ' - id: foo\n' + '- repo: {path}\n' + " rev: '{rev}'\n" + ' hooks:\n' + ' - id: foo\n' + ) + cfg = tmpdir.join(C.CONFIG_FILE) + cfg.write(fmt.format(path=out_of_date.path, rev=out_of_date.original_rev)) + + assert autoupdate(str(cfg), freeze=False, tags_only=False) == 0 + expected = fmt.format(path=out_of_date.path, rev=out_of_date.head_rev) + assert cfg.read() == expected diff --git a/tests/commands/clean_test.py b/tests/commands/clean_test.py index dc33ebb0..dd8e4a53 100644 --- a/tests/commands/clean_test.py +++ b/tests/commands/clean_test.py @@ -1,8 +1,8 @@ -from __future__ import unicode_literals +from __future__ import annotations import os.path +from unittest import mock -import mock import pytest from pre_commit.commands.clean import clean diff --git a/tests/commands/gc_test.py b/tests/commands/gc_test.py index 5be86b1b..992b02f3 100644 --- a/tests/commands/gc_test.py +++ b/tests/commands/gc_test.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import os import pre_commit.constants as C @@ -17,11 +19,13 @@ from testing.util import git_commit def _repo_count(store): - return len(store.select_all_repos()) + with store.connect() as db: + return db.execute('SELECT COUNT(1) FROM repos').fetchone()[0] def _config_count(store): - return len(store.select_all_configs()) + with store.connect() as db: + return db.execute('SELECT COUNT(1) FROM configs').fetchone()[0] def _remove_config_assert_cleared(store, cap_out): @@ -41,8 +45,9 @@ def test_gc(tempdir_factory, store, in_git_dir, cap_out): store.mark_config_used(C.CONFIG_FILE) # update will clone both the old and new repo, making the old one gc-able - install_hooks(C.CONFIG_FILE, store) - assert not autoupdate(C.CONFIG_FILE, store, tags_only=False) + assert not install_hooks(C.CONFIG_FILE, store) + assert not autoupdate(C.CONFIG_FILE, freeze=False, tags_only=False) + assert not install_hooks(C.CONFIG_FILE, store) assert _config_count(store) == 1 assert _repo_count(store) == 2 @@ -150,7 +155,8 @@ def test_invalid_manifest_gcd(tempdir_factory, store, in_git_dir, cap_out): install_hooks(C.CONFIG_FILE, store) # we'll "break" the manifest to simulate an old version clone - (_, _, path), = store.select_all_repos() + with store.connect() as db: + path, = db.execute('SELECT path FROM repos').fetchone() os.remove(os.path.join(path, C.MANIFEST_FILE)) assert _config_count(store) == 1 @@ -159,3 +165,11 @@ def test_invalid_manifest_gcd(tempdir_factory, store, in_git_dir, cap_out): assert _config_count(store) == 1 assert _repo_count(store) == 0 assert cap_out.get().splitlines()[-1] == '1 repo(s) removed.' + + +def test_gc_pre_1_14_roll_forward(store, cap_out): + with store.connect() as db: # simulate pre-1.14.0 + db.executescript('DROP TABLE configs') + + assert not gc(store) + assert cap_out.get() == '0 repo(s) removed.\n' diff --git a/tests/commands/hazmat_test.py b/tests/commands/hazmat_test.py new file mode 100644 index 00000000..df957e36 --- /dev/null +++ b/tests/commands/hazmat_test.py @@ -0,0 +1,99 @@ +from __future__ import annotations + +import sys + +import pytest + +from pre_commit.commands.hazmat import _cmd_filenames +from pre_commit.commands.hazmat import main +from testing.util import cwd + + +def test_cmd_filenames_no_dash_dash(): + with pytest.raises(SystemExit) as excinfo: + _cmd_filenames(('no', 'dashdash', 'here')) + msg, = excinfo.value.args + assert msg == 'hazmat entry must end with `--`' + + +def test_cmd_filenames_no_filenames(): + cmd, filenames = _cmd_filenames(('hello', 'world', '--')) + assert cmd == ('hello', 'world') + assert filenames == () + + +def test_cmd_filenames_some_filenames(): + cmd, filenames = _cmd_filenames(('hello', 'world', '--', 'f1', 'f2')) + assert cmd == ('hello', 'world') + assert filenames == ('f1', 'f2') + + +def test_cmd_filenames_multiple_dashdash(): + cmd, filenames = _cmd_filenames(('hello', '--', 'arg', '--', 'f1', 'f2')) + assert cmd == ('hello', '--', 'arg') + assert filenames == ('f1', 'f2') + + +def test_cd_unexpected_filename(): + with pytest.raises(SystemExit) as excinfo: + main(('cd', 'subdir', 'cmd', '--', 'subdir/1', 'not-subdir/2')) + msg, = excinfo.value.args + assert msg == "unexpected file without prefix='subdir/': not-subdir/2" + + +def _norm(out): + return out.replace('\r\n', '\n') + + +def test_cd(tmp_path, capfd): + subdir = tmp_path.joinpath('subdir') + subdir.mkdir() + subdir.joinpath('a').write_text('a') + subdir.joinpath('b').write_text('b') + + with cwd(tmp_path): + ret = main(( + 'cd', 'subdir', + sys.executable, '-c', + 'import os; print(os.getcwd());' + 'import sys; [print(open(f).read()) for f in sys.argv[1:]]', + '--', + 'subdir/a', 'subdir/b', + )) + + assert ret == 0 + out, err = capfd.readouterr() + assert _norm(out) == f'{subdir}\na\nb\n' + assert err == '' + + +def test_ignore_exit_code(capfd): + ret = main(( + 'ignore-exit-code', sys.executable, '-c', 'raise SystemExit("bye")', + )) + assert ret == 0 + out, err = capfd.readouterr() + assert out == '' + assert _norm(err) == 'bye\n' + + +def test_n1(capfd): + ret = main(( + 'n1', sys.executable, '-c', 'import sys; print(sys.argv[1:])', + '--', + 'foo', 'bar', 'baz', + )) + assert ret == 0 + out, err = capfd.readouterr() + assert _norm(out) == "['foo']\n['bar']\n['baz']\n" + assert err == '' + + +def test_n1_some_error_code(): + ret = main(( + 'n1', sys.executable, '-c', + 'import sys; raise SystemExit(sys.argv[1] == "error")', + '--', + 'ok', 'error', 'ok', + )) + assert ret == 1 diff --git a/tests/commands/hook_impl_test.py b/tests/commands/hook_impl_test.py new file mode 100644 index 00000000..d757e85c --- /dev/null +++ b/tests/commands/hook_impl_test.py @@ -0,0 +1,381 @@ +from __future__ import annotations + +import subprocess +import sys +from unittest import mock + +import pytest + +import pre_commit.constants as C +from pre_commit import git +from pre_commit.commands import hook_impl +from pre_commit.envcontext import envcontext +from pre_commit.util import cmd_output +from pre_commit.util import make_executable +from testing.fixtures import git_dir +from testing.fixtures import sample_local_config +from testing.fixtures import write_config +from testing.util import cwd +from testing.util import git_commit + + +def test_validate_config_file_exists(tmpdir): + cfg = tmpdir.join(C.CONFIG_FILE).ensure() + hook_impl._validate_config(0, cfg, True) + + +def test_validate_config_missing(capsys): + with pytest.raises(SystemExit) as excinfo: + hook_impl._validate_config(123, 'DNE.yaml', False) + ret, = excinfo.value.args + assert ret == 1 + assert capsys.readouterr().out == ( + 'No DNE.yaml file was found\n' + '- To temporarily silence this, run ' + '`PRE_COMMIT_ALLOW_NO_CONFIG=1 git ...`\n' + '- To permanently silence this, install pre-commit with the ' + '--allow-missing-config option\n' + '- To uninstall pre-commit run `pre-commit uninstall`\n' + ) + + +def test_validate_config_skip_missing_config(capsys): + with pytest.raises(SystemExit) as excinfo: + hook_impl._validate_config(123, 'DNE.yaml', True) + ret, = excinfo.value.args + assert ret == 123 + expected = '`DNE.yaml` config file not found. Skipping `pre-commit`.\n' + assert capsys.readouterr().out == expected + + +def test_validate_config_skip_via_env_variable(capsys): + with pytest.raises(SystemExit) as excinfo: + with envcontext((('PRE_COMMIT_ALLOW_NO_CONFIG', '1'),)): + hook_impl._validate_config(0, 'DNE.yaml', False) + ret, = excinfo.value.args + assert ret == 0 + expected = '`DNE.yaml` config file not found. Skipping `pre-commit`.\n' + assert capsys.readouterr().out == expected + + +def test_run_legacy_does_not_exist(tmpdir): + retv, stdin = hook_impl._run_legacy('pre-commit', tmpdir, ()) + assert (retv, stdin) == (0, b'') + + +def test_run_legacy_executes_legacy_script(tmpdir, capfd): + hook = tmpdir.join('pre-commit.legacy') + hook.write('#!/usr/bin/env bash\necho hi "$@"\nexit 1\n') + make_executable(hook) + retv, stdin = hook_impl._run_legacy('pre-commit', tmpdir, ('arg1', 'arg2')) + assert capfd.readouterr().out.strip() == 'hi arg1 arg2' + assert (retv, stdin) == (1, b'') + + +def test_run_legacy_pre_push_returns_stdin(tmpdir): + with mock.patch.object(sys.stdin.buffer, 'read', return_value=b'stdin'): + retv, stdin = hook_impl._run_legacy('pre-push', tmpdir, ()) + assert (retv, stdin) == (0, b'stdin') + + +def test_run_legacy_recursive(tmpdir): + hook = tmpdir.join('pre-commit.legacy').ensure() + make_executable(hook) + + # simulate a call being recursive + def call(*_, **__): + return hook_impl._run_legacy('pre-commit', tmpdir, ()) + + with mock.patch.object(subprocess, 'run', call): + with pytest.raises(SystemExit): + call() + + +@pytest.mark.parametrize( + ('hook_type', 'args'), + ( + ('pre-commit', []), + ('pre-merge-commit', []), + ('pre-push', ['branch_name', 'remote_name']), + ('commit-msg', ['.git/COMMIT_EDITMSG']), + ('post-commit', []), + ('post-merge', ['1']), + ('pre-rebase', ['main', 'topic']), + ('pre-rebase', ['main']), + ('post-checkout', ['old_head', 'new_head', '1']), + ('post-rewrite', ['amend']), + # multiple choices for commit-editmsg + ('prepare-commit-msg', ['.git/COMMIT_EDITMSG']), + ('prepare-commit-msg', ['.git/COMMIT_EDITMSG', 'message']), + ('prepare-commit-msg', ['.git/COMMIT_EDITMSG', 'commit', 'deadbeef']), + ), +) +def test_check_args_length_ok(hook_type, args): + hook_impl._check_args_length(hook_type, args) + + +def test_check_args_length_error_too_many_plural(): + with pytest.raises(SystemExit) as excinfo: + hook_impl._check_args_length('pre-commit', ['run', '--all-files']) + msg, = excinfo.value.args + assert msg == ( + 'hook-impl for pre-commit expected 0 arguments but got 2: ' + "['run', '--all-files']" + ) + + +def test_check_args_length_error_too_many_singular(): + with pytest.raises(SystemExit) as excinfo: + hook_impl._check_args_length('commit-msg', []) + msg, = excinfo.value.args + assert msg == 'hook-impl for commit-msg expected 1 argument but got 0: []' + + +def test_check_args_length_prepare_commit_msg_error(): + with pytest.raises(SystemExit) as excinfo: + hook_impl._check_args_length('prepare-commit-msg', []) + msg, = excinfo.value.args + assert msg == ( + 'hook-impl for prepare-commit-msg expected 1, 2, or 3 arguments ' + 'but got 0: []' + ) + + +def test_check_args_length_pre_rebase_error(): + with pytest.raises(SystemExit) as excinfo: + hook_impl._check_args_length('pre-rebase', []) + msg, = excinfo.value.args + assert msg == 'hook-impl for pre-rebase expected 1 or 2 arguments but got 0: []' # noqa: E501 + + +def test_run_ns_pre_commit(): + ns = hook_impl._run_ns('pre-commit', True, (), b'') + assert ns is not None + assert ns.hook_stage == 'pre-commit' + assert ns.color is True + + +def test_run_ns_pre_rebase(): + ns = hook_impl._run_ns('pre-rebase', True, ('main', 'topic'), b'') + assert ns is not None + assert ns.hook_stage == 'pre-rebase' + assert ns.color is True + assert ns.pre_rebase_upstream == 'main' + assert ns.pre_rebase_branch == 'topic' + + ns = hook_impl._run_ns('pre-rebase', True, ('main',), b'') + assert ns is not None + assert ns.hook_stage == 'pre-rebase' + assert ns.color is True + assert ns.pre_rebase_upstream == 'main' + assert ns.pre_rebase_branch is None + + +def test_run_ns_commit_msg(): + ns = hook_impl._run_ns('commit-msg', False, ('.git/COMMIT_MSG',), b'') + assert ns is not None + assert ns.hook_stage == 'commit-msg' + assert ns.color is False + assert ns.commit_msg_filename == '.git/COMMIT_MSG' + + +def test_run_ns_prepare_commit_msg_one_arg(): + ns = hook_impl._run_ns( + 'prepare-commit-msg', False, + ('.git/COMMIT_MSG',), b'', + ) + assert ns is not None + assert ns.hook_stage == 'prepare-commit-msg' + assert ns.color is False + assert ns.commit_msg_filename == '.git/COMMIT_MSG' + + +def test_run_ns_prepare_commit_msg_two_arg(): + ns = hook_impl._run_ns( + 'prepare-commit-msg', False, + ('.git/COMMIT_MSG', 'message'), b'', + ) + assert ns is not None + assert ns.hook_stage == 'prepare-commit-msg' + assert ns.color is False + assert ns.commit_msg_filename == '.git/COMMIT_MSG' + assert ns.prepare_commit_message_source == 'message' + + +def test_run_ns_prepare_commit_msg_three_arg(): + ns = hook_impl._run_ns( + 'prepare-commit-msg', False, + ('.git/COMMIT_MSG', 'message', 'HEAD'), b'', + ) + assert ns is not None + assert ns.hook_stage == 'prepare-commit-msg' + assert ns.color is False + assert ns.commit_msg_filename == '.git/COMMIT_MSG' + assert ns.prepare_commit_message_source == 'message' + assert ns.commit_object_name == 'HEAD' + + +def test_run_ns_post_commit(): + ns = hook_impl._run_ns('post-commit', True, (), b'') + assert ns is not None + assert ns.hook_stage == 'post-commit' + assert ns.color is True + + +def test_run_ns_post_merge(): + ns = hook_impl._run_ns('post-merge', True, ('1',), b'') + assert ns is not None + assert ns.hook_stage == 'post-merge' + assert ns.color is True + assert ns.is_squash_merge == '1' + + +def test_run_ns_post_rewrite(): + ns = hook_impl._run_ns('post-rewrite', True, ('amend',), b'') + assert ns is not None + assert ns.hook_stage == 'post-rewrite' + assert ns.color is True + assert ns.rewrite_command == 'amend' + + +def test_run_ns_post_checkout(): + ns = hook_impl._run_ns('post-checkout', True, ('a', 'b', 'c'), b'') + assert ns is not None + assert ns.hook_stage == 'post-checkout' + assert ns.color is True + assert ns.from_ref == 'a' + assert ns.to_ref == 'b' + assert ns.checkout_type == 'c' + + +@pytest.fixture +def push_example(tempdir_factory): + src = git_dir(tempdir_factory) + git_commit(cwd=src) + src_head = git.head_rev(src) + + clone = tempdir_factory.get() + cmd_output('git', 'clone', src, clone) + git_commit(cwd=clone) + clone_head = git.head_rev(clone) + return (src, src_head, clone, clone_head) + + +def test_run_ns_pre_push_updating_branch(push_example): + src, src_head, clone, clone_head = push_example + + with cwd(clone): + args = ('origin', src) + stdin = f'HEAD {clone_head} refs/heads/b {src_head}\n'.encode() + ns = hook_impl._run_ns('pre-push', False, args, stdin) + + assert ns is not None + assert ns.hook_stage == 'pre-push' + assert ns.color is False + assert ns.remote_name == 'origin' + assert ns.remote_url == src + assert ns.from_ref == src_head + assert ns.to_ref == clone_head + assert ns.all_files is False + + +def test_run_ns_pre_push_new_branch(push_example): + src, src_head, clone, clone_head = push_example + + with cwd(clone): + args = ('origin', src) + stdin = f'HEAD {clone_head} refs/heads/b {hook_impl.Z40}\n'.encode() + ns = hook_impl._run_ns('pre-push', False, args, stdin) + + assert ns is not None + assert ns.from_ref == src_head + assert ns.to_ref == clone_head + + +def test_run_ns_pre_push_new_branch_existing_rev(push_example): + src, src_head, clone, _ = push_example + + with cwd(clone): + args = ('origin', src) + stdin = f'HEAD {src_head} refs/heads/b2 {hook_impl.Z40}\n'.encode() + ns = hook_impl._run_ns('pre-push', False, args, stdin) + + assert ns is None + + +def test_run_ns_pre_push_ref_with_whitespace(push_example): + src, src_head, clone, _ = push_example + + with cwd(clone): + args = ('origin', src) + line = f'HEAD^{{/ }} {src_head} refs/heads/b2 {hook_impl.Z40}\n' + stdin = line.encode() + ns = hook_impl._run_ns('pre-push', False, args, stdin) + + assert ns is None + + +def test_pushing_orphan_branch(push_example): + src, src_head, clone, _ = push_example + + cmd_output('git', 'checkout', '--orphan', 'b2', cwd=clone) + git_commit(cwd=clone, msg='something else to get unique hash') + clone_rev = git.head_rev(clone) + + with cwd(clone): + args = ('origin', src) + stdin = f'HEAD {clone_rev} refs/heads/b2 {hook_impl.Z40}\n'.encode() + ns = hook_impl._run_ns('pre-push', False, args, stdin) + + assert ns is not None + assert ns.all_files is True + + +def test_run_ns_pre_push_deleting_branch(push_example): + src, src_head, clone, _ = push_example + + with cwd(clone): + args = ('origin', src) + stdin = f'(delete) {hook_impl.Z40} refs/heads/b {src_head}'.encode() + ns = hook_impl._run_ns('pre-push', False, args, stdin) + + assert ns is None + + +def test_hook_impl_main_noop_pre_push(cap_out, store, push_example): + src, src_head, clone, _ = push_example + + stdin = f'(delete) {hook_impl.Z40} refs/heads/b {src_head}'.encode() + with mock.patch.object(sys.stdin.buffer, 'read', return_value=stdin): + with cwd(clone): + write_config('.', sample_local_config()) + ret = hook_impl.hook_impl( + store, + config=C.CONFIG_FILE, + color=False, + hook_type='pre-push', + hook_dir='.git/hooks', + skip_on_missing_config=False, + args=('origin', src), + ) + assert ret == 0 + assert cap_out.get() == '' + + +def test_hook_impl_main_runs_hooks(cap_out, tempdir_factory, store): + with cwd(git_dir(tempdir_factory)): + write_config('.', sample_local_config()) + ret = hook_impl.hook_impl( + store, + config=C.CONFIG_FILE, + color=False, + hook_type='pre-commit', + hook_dir='.git/hooks', + skip_on_missing_config=False, + args=(), + ) + assert ret == 0 + expected = '''\ +Block if "DO NOT COMMIT" is found....................(no files to check)Skipped +''' + assert cap_out.get() == expected diff --git a/tests/commands/init_templatedir_test.py b/tests/commands/init_templatedir_test.py index b94de99a..28f29b77 100644 --- a/tests/commands/init_templatedir_test.py +++ b/tests/commands/init_templatedir_test.py @@ -1,7 +1,9 @@ -import os.path -import subprocess +from __future__ import annotations -import mock +import os.path +from unittest import mock + +import pytest import pre_commit.constants as C from pre_commit.commands.init_templatedir import init_templatedir @@ -16,7 +18,7 @@ from testing.util import git_commit def test_init_templatedir(tmpdir, tempdir_factory, store, cap_out): target = str(tmpdir.join('tmpl')) - init_templatedir(C.CONFIG_FILE, store, target, hook_type='pre-commit') + init_templatedir(C.CONFIG_FILE, store, target, hook_types=['pre-commit']) lines = cap_out.get().splitlines() assert lines[0].startswith('pre-commit installed at ') assert lines[1] == ( @@ -26,15 +28,13 @@ def test_init_templatedir(tmpdir, tempdir_factory, store, cap_out): '[WARNING] maybe `git config --global init.templateDir', ) - with envcontext([('GIT_TEMPLATE_DIR', target)]): + with envcontext((('GIT_TEMPLATE_DIR', target),)): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - retcode, output, _ = git_commit( + retcode, output = git_commit( fn=cmd_output_mocked_pre_commit_home, tempdir_factory=tempdir_factory, - # git commit puts pre-commit to stderr - stderr=subprocess.STDOUT, ) assert retcode == 0 assert 'Bash hook....' in output @@ -45,7 +45,9 @@ def test_init_templatedir_already_set(tmpdir, tempdir_factory, store, cap_out): tmp_git_dir = git_dir(tempdir_factory) with cwd(tmp_git_dir): cmd_output('git', 'config', 'init.templateDir', target) - init_templatedir(C.CONFIG_FILE, store, target, hook_type='pre-commit') + init_templatedir( + C.CONFIG_FILE, store, target, hook_types=['pre-commit'], + ) lines = cap_out.get().splitlines() assert len(lines) == 1 @@ -54,10 +56,12 @@ def test_init_templatedir_already_set(tmpdir, tempdir_factory, store, cap_out): def test_init_templatedir_not_set(tmpdir, store, cap_out): # set HOME to ignore the current `.gitconfig` - with envcontext([('HOME', str(tmpdir))]): + with envcontext((('HOME', str(tmpdir)),)): with tmpdir.join('tmpl').ensure_dir().as_cwd(): # we have not set init.templateDir so this should produce a warning - init_templatedir(C.CONFIG_FILE, store, '.', hook_type='pre-commit') + init_templatedir( + C.CONFIG_FILE, store, '.', hook_types=['pre-commit'], + ) lines = cap_out.get().splitlines() assert len(lines) == 3 @@ -73,9 +77,66 @@ def test_init_templatedir_expanduser(tmpdir, tempdir_factory, store, cap_out): cmd_output('git', 'config', 'init.templateDir', '~/templatedir') with mock.patch.object(os.path, 'expanduser', return_value=target): init_templatedir( - C.CONFIG_FILE, store, target, hook_type='pre-commit', + C.CONFIG_FILE, store, target, hook_types=['pre-commit'], ) lines = cap_out.get().splitlines() assert len(lines) == 1 assert lines[0].startswith('pre-commit installed at') + + +def test_init_templatedir_hookspath_set(tmpdir, tempdir_factory, store): + target = tmpdir.join('tmpl') + tmp_git_dir = git_dir(tempdir_factory) + with cwd(tmp_git_dir): + cmd_output('git', 'config', '--local', 'core.hooksPath', 'hooks') + init_templatedir( + C.CONFIG_FILE, store, target, hook_types=['pre-commit'], + ) + assert target.join('hooks/pre-commit').exists() + + +@pytest.mark.parametrize( + ('skip', 'commit_retcode', 'commit_output_snippet'), + ( + (True, 0, 'Skipping `pre-commit`.'), + (False, 1, f'No {C.CONFIG_FILE} file was found'), + ), +) +def test_init_templatedir_skip_on_missing_config( + tmpdir, + tempdir_factory, + store, + cap_out, + skip, + commit_retcode, + commit_output_snippet, +): + target = str(tmpdir.join('tmpl')) + init_git_dir = git_dir(tempdir_factory) + with cwd(init_git_dir): + cmd_output('git', 'config', 'init.templateDir', target) + init_templatedir( + C.CONFIG_FILE, + store, + target, + hook_types=['pre-commit'], + skip_on_missing_config=skip, + ) + + lines = cap_out.get().splitlines() + assert len(lines) == 1 + assert lines[0].startswith('pre-commit installed at') + + with envcontext((('GIT_TEMPLATE_DIR', target),)): + verify_git_dir = git_dir(tempdir_factory) + + with cwd(verify_git_dir): + retcode, output = git_commit( + fn=cmd_output_mocked_pre_commit_home, + tempdir_factory=tempdir_factory, + check=False, + ) + + assert retcode == commit_retcode + assert commit_output_snippet in output diff --git a/tests/commands/install_uninstall_test.py b/tests/commands/install_uninstall_test.py index 913bf74e..9eb0e741 100644 --- a/tests/commands/install_uninstall_test.py +++ b/tests/commands/install_uninstall_test.py @@ -1,35 +1,61 @@ -# -*- coding: UTF-8 -*- -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import io import os.path import re -import subprocess -import sys -import mock +import re_assert import pre_commit.constants as C +from pre_commit import git +from pre_commit.commands.install_uninstall import _hook_types from pre_commit.commands.install_uninstall import CURRENT_HASH from pre_commit.commands.install_uninstall import install from pre_commit.commands.install_uninstall import install_hooks from pre_commit.commands.install_uninstall import is_our_script from pre_commit.commands.install_uninstall import PRIOR_HASHES -from pre_commit.commands.install_uninstall import shebang from pre_commit.commands.install_uninstall import uninstall from pre_commit.parse_shebang import find_executable from pre_commit.util import cmd_output from pre_commit.util import make_executable -from pre_commit.util import mkdirp from pre_commit.util import resource_text +from testing.fixtures import add_config_to_repo from testing.fixtures import git_dir from testing.fixtures import make_consuming_repo from testing.fixtures import remove_config_from_repo +from testing.fixtures import write_config from testing.util import cmd_output_mocked_pre_commit_home from testing.util import cwd from testing.util import git_commit -from testing.util import xfailif_no_symlink + + +def test_hook_types_explicitly_listed(): + assert _hook_types(os.devnull, ['pre-push']) == ['pre-push'] + + +def test_hook_types_default_value_when_not_specified(): + assert _hook_types(os.devnull, None) == ['pre-commit'] + + +def test_hook_types_configured(tmpdir): + cfg = tmpdir.join('t.cfg') + cfg.write('default_install_hook_types: [pre-push]\nrepos: []\n') + + assert _hook_types(str(cfg), None) == ['pre-push'] + + +def test_hook_types_configured_nonsense(tmpdir): + cfg = tmpdir.join('t.cfg') + cfg.write('default_install_hook_types: []\nrepos: []\n') + + # hopefully the user doesn't do this, but the code allows it! + assert _hook_types(str(cfg), None) == [] + + +def test_hook_types_configuration_has_error(tmpdir): + cfg = tmpdir.join('t.cfg') + cfg.write('[') + + assert _hook_types(str(cfg), None) == ['pre-commit'] def test_is_not_script(): @@ -42,35 +68,15 @@ def test_is_script(): def test_is_previous_pre_commit(tmpdir): f = tmpdir.join('foo') - f.write(PRIOR_HASHES[0] + '\n') + f.write(f'{PRIOR_HASHES[0].decode()}\n') assert is_our_script(f.strpath) -def test_shebang_windows(): - with mock.patch.object(sys, 'platform', 'win32'): - assert shebang() == '#!/usr/bin/env python' - - -def test_shebang_posix_not_on_path(): - with mock.patch.object(sys, 'platform', 'posix'): - with mock.patch.object(os, 'defpath', ''): - assert shebang() == '#!/usr/bin/env python' - - -def test_shebang_posix_on_path(tmpdir): - tmpdir.join('python{}'.format(sys.version_info[0])).ensure() - - with mock.patch.object(sys, 'platform', 'posix'): - with mock.patch.object(os, 'defpath', tmpdir.strpath): - expected = '#!/usr/bin/env python{}'.format(sys.version_info[0]) - assert shebang() == expected - - def test_install_pre_commit(in_git_dir, store): - assert not install(C.CONFIG_FILE, store) + assert not install(C.CONFIG_FILE, store, hook_types=['pre-commit']) assert os.access(in_git_dir.join('.git/hooks/pre-commit').strpath, os.X_OK) - assert not install(C.CONFIG_FILE, store, hook_type='pre-push') + assert not install(C.CONFIG_FILE, store, hook_types=['pre-push']) assert os.access(in_git_dir.join('.git/hooks/pre-push').strpath, os.X_OK) @@ -78,32 +84,40 @@ def test_install_hooks_directory_not_present(in_git_dir, store): # Simulate some git clients which don't make .git/hooks #234 if in_git_dir.join('.git/hooks').exists(): # pragma: no cover (odd git) in_git_dir.join('.git/hooks').remove() - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) assert in_git_dir.join('.git/hooks/pre-commit').exists() +def test_install_multiple_hooks_at_once(in_git_dir, store): + install(C.CONFIG_FILE, store, hook_types=['pre-commit', 'pre-push']) + assert in_git_dir.join('.git/hooks/pre-commit').exists() + assert in_git_dir.join('.git/hooks/pre-push').exists() + uninstall(C.CONFIG_FILE, hook_types=['pre-commit', 'pre-push']) + assert not in_git_dir.join('.git/hooks/pre-commit').exists() + assert not in_git_dir.join('.git/hooks/pre-push').exists() + + def test_install_refuses_core_hookspath(in_git_dir, store): cmd_output('git', 'config', '--local', 'core.hooksPath', 'hooks') - assert install(C.CONFIG_FILE, store) + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) -@xfailif_no_symlink # pragma: windows no cover def test_install_hooks_dead_symlink(in_git_dir, store): hook = in_git_dir.join('.git/hooks').ensure_dir().join('pre-commit') os.symlink('/fake/baz', hook.strpath) - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) assert hook.exists() def test_uninstall_does_not_blow_up_when_not_there(in_git_dir): - assert uninstall() == 0 + assert uninstall(C.CONFIG_FILE, hook_types=['pre-commit']) == 0 def test_uninstall(in_git_dir, store): assert not in_git_dir.join('.git/hooks/pre-commit').exists() - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) assert in_git_dir.join('.git/hooks/pre-commit').exists() - uninstall() + uninstall(C.CONFIG_FILE, hook_types=['pre-commit']) assert not in_git_dir.join('.git/hooks/pre-commit').exists() @@ -112,53 +126,51 @@ def _get_commit_output(tempdir_factory, touch_file='foo', **kwargs): cmd_output('git', 'add', touch_file) return git_commit( fn=cmd_output_mocked_pre_commit_home, - # git commit puts pre-commit to stderr - stderr=subprocess.STDOUT, - retcode=None, + check=False, tempdir_factory=tempdir_factory, - **kwargs - )[:2] + **kwargs, + ) # osx does this different :( FILES_CHANGED = ( r'(' - r' 1 file changed, 0 insertions\(\+\), 0 deletions\(-\)\r?\n' + r' 1 file changed, 0 insertions\(\+\), 0 deletions\(-\)\n' r'|' - r' 0 files changed\r?\n' + r' 0 files changed\n' r')' ) -NORMAL_PRE_COMMIT_RUN = re.compile( - r'^\[INFO\] Initializing environment for .+\.\r?\n' - r'Bash hook\.+Passed\r?\n' - r'\[master [a-f0-9]{7}\] commit!\r?\n' + - FILES_CHANGED + - r' create mode 100644 foo\r?\n$', +NORMAL_PRE_COMMIT_RUN = re_assert.Matches( + fr'^\[INFO\] Initializing environment for .+\.\n' + fr'Bash hook\.+Passed\n' + fr'\[master [a-f0-9]{{7}}\] commit!\n' + fr'{FILES_CHANGED}' + fr' create mode 100644 foo\n$', ) def test_install_pre_commit_and_run(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_install_pre_commit_and_run_custom_path(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - cmd_output('git', 'mv', C.CONFIG_FILE, 'custom-config.yaml') + cmd_output('git', 'mv', C.CONFIG_FILE, 'custom.yaml') git_commit(cwd=path) - assert install('custom-config.yaml', store) == 0 + assert install('custom.yaml', store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_install_in_submodule_and_run(tempdir_factory, store): @@ -169,10 +181,10 @@ def test_install_in_submodule_and_run(tempdir_factory, store): sub_pth = os.path.join(parent_path, 'sub') with cwd(sub_pth): - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_install_in_worktree_and_run(tempdir_factory, store): @@ -182,10 +194,10 @@ def test_install_in_worktree_and_run(tempdir_factory, store): cmd_output('git', '-C', src_path, 'worktree', 'add', path, '-b', 'master') with cwd(path): - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_commit_am(tempdir_factory, store): @@ -196,10 +208,10 @@ def test_commit_am(tempdir_factory, store): open('unstaged', 'w').close() cmd_output('git', 'add', '.') git_commit(cwd=path) - with io.open('unstaged', 'w') as foo_file: + with open('unstaged', 'w') as foo_file: foo_file.write('Oh hai') - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 @@ -208,7 +220,7 @@ def test_commit_am(tempdir_factory, store): def test_unicode_merge_commit_message(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 cmd_output('git', 'checkout', 'master', '-b', 'foo') git_commit('-n', cwd=path) cmd_output('git', 'checkout', 'master') @@ -225,90 +237,99 @@ def test_unicode_merge_commit_message(tempdir_factory, store): def test_install_idempotent(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - assert install(C.CONFIG_FILE, store) == 0 - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def _path_without_us(): # Choose a path which *probably* doesn't include us env = dict(os.environ) - exe = find_executable('pre-commit', _environ=env) + exe = find_executable('pre-commit', env=env) while exe: parts = env['PATH'].split(os.pathsep) - after = [x for x in parts if x.lower() != os.path.dirname(exe).lower()] + after = [ + x for x in parts + if x.lower().rstrip(os.sep) != os.path.dirname(exe).lower() + ] if parts == after: raise AssertionError(exe, parts) env['PATH'] = os.pathsep.join(after) - exe = find_executable('pre-commit', _environ=env) + exe = find_executable('pre-commit', env=env) return env['PATH'] def test_environment_not_sourced(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - # Patch the executable to simulate rming virtualenv - with mock.patch.object(sys, 'executable', '/does-not-exist'): - assert install(C.CONFIG_FILE, store) == 0 + assert not install(C.CONFIG_FILE, store, hook_types=['pre-commit']) + # simulate deleting the virtualenv by rewriting the exe + hook = os.path.join(path, '.git/hooks/pre-commit') + with open(hook) as f: + src = f.read() + src = re.sub('\nINSTALL_PYTHON=.*\n', '\nINSTALL_PYTHON="/dne"\n', src) + with open(hook, 'w') as f: + f.write(src) # Use a specific homedir to ignore --user installs homedir = tempdir_factory.get() - ret, stdout, stderr = git_commit( - env={ - 'HOME': homedir, - 'PATH': _path_without_us(), - # Git needs this to make a commit - 'GIT_AUTHOR_NAME': os.environ['GIT_AUTHOR_NAME'], - 'GIT_COMMITTER_NAME': os.environ['GIT_COMMITTER_NAME'], - 'GIT_AUTHOR_EMAIL': os.environ['GIT_AUTHOR_EMAIL'], - 'GIT_COMMITTER_EMAIL': os.environ['GIT_COMMITTER_EMAIL'], - }, - retcode=None, - ) + env = { + 'HOME': homedir, + 'PATH': _path_without_us(), + # Git needs this to make a commit + 'GIT_AUTHOR_NAME': os.environ['GIT_AUTHOR_NAME'], + 'GIT_COMMITTER_NAME': os.environ['GIT_COMMITTER_NAME'], + 'GIT_AUTHOR_EMAIL': os.environ['GIT_AUTHOR_EMAIL'], + 'GIT_COMMITTER_EMAIL': os.environ['GIT_COMMITTER_EMAIL'], + } + if os.name == 'nt' and 'PATHEXT' in os.environ: # pragma: no cover + env['PATHEXT'] = os.environ['PATHEXT'] + + ret, out = git_commit(env=env, check=False) assert ret == 1 - assert stdout == '' - assert stderr.replace('\r\n', '\n') == ( + assert out == ( '`pre-commit` not found. ' 'Did you forget to activate your virtualenv?\n' ) -FAILING_PRE_COMMIT_RUN = re.compile( - r'^\[INFO\] Initializing environment for .+\.\r?\n' - r'Failing hook\.+Failed\r?\n' - r'hookid: failing_hook\r?\n' - r'\r?\n' - r'Fail\r?\n' - r'foo\r?\n' - r'\r?\n$', +FAILING_PRE_COMMIT_RUN = re_assert.Matches( + r'^\[INFO\] Initializing environment for .+\.\n' + r'Failing hook\.+Failed\n' + r'- hook id: failing_hook\n' + r'- exit code: 1\n' + r'\n' + r'Fail\n' + r'foo\n' + r'\n$', ) def test_failing_hooks_returns_nonzero(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'failing_hook_repo') with cwd(path): - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 1 - assert FAILING_PRE_COMMIT_RUN.match(output) + FAILING_PRE_COMMIT_RUN.assert_matches(output) -EXISTING_COMMIT_RUN = re.compile( - r'^legacy hook\r?\n' - r'\[master [a-f0-9]{7}\] commit!\r?\n' + - FILES_CHANGED + - r' create mode 100644 baz\r?\n$', +EXISTING_COMMIT_RUN = re_assert.Matches( + fr'^legacy hook\n' + fr'\[master [a-f0-9]{{7}}\] commit!\n' + fr'{FILES_CHANGED}' + fr' create mode 100644 baz\n$', ) def _write_legacy_hook(path): - mkdirp(os.path.join(path, '.git/hooks')) - with io.open(os.path.join(path, '.git/hooks/pre-commit'), 'w') as f: - f.write('#!/usr/bin/env bash\necho "legacy hook"\n') + os.makedirs(os.path.join(path, '.git/hooks'), exist_ok=True) + with open(os.path.join(path, '.git/hooks/pre-commit'), 'w') as f: + f.write('#!/usr/bin/env bash\necho legacy hook\n') make_executable(f.name) @@ -320,26 +341,27 @@ def test_install_existing_hooks_no_overwrite(tempdir_factory, store): # Make sure we installed the "old" hook correctly ret, output = _get_commit_output(tempdir_factory, touch_file='baz') assert ret == 0 - assert EXISTING_COMMIT_RUN.match(output) + EXISTING_COMMIT_RUN.assert_matches(output) # Now install pre-commit (no-overwrite) - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 # We should run both the legacy and pre-commit hooks ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert output.startswith('legacy hook\n') - assert NORMAL_PRE_COMMIT_RUN.match(output[len('legacy hook\n'):]) + legacy = 'legacy hook\n' + assert output.startswith(legacy) + NORMAL_PRE_COMMIT_RUN.assert_matches(output.removeprefix(legacy)) def test_legacy_overwriting_legacy_hook(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): _write_legacy_hook(path) - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 _write_legacy_hook(path) # this previously crashed on windows. See #1010 - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 def test_install_existing_hook_no_overwrite_idempotent(tempdir_factory, store): @@ -348,20 +370,34 @@ def test_install_existing_hook_no_overwrite_idempotent(tempdir_factory, store): _write_legacy_hook(path) # Install twice - assert install(C.CONFIG_FILE, store) == 0 - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 # We should run both the legacy and pre-commit hooks ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert output.startswith('legacy hook\n') - assert NORMAL_PRE_COMMIT_RUN.match(output[len('legacy hook\n'):]) + legacy = 'legacy hook\n' + assert output.startswith(legacy) + NORMAL_PRE_COMMIT_RUN.assert_matches(output.removeprefix(legacy)) -FAIL_OLD_HOOK = re.compile( - r'fail!\r?\n' - r'\[INFO\] Initializing environment for .+\.\r?\n' - r'Bash hook\.+Passed\r?\n', +def test_install_with_existing_non_utf8_script(tmpdir, store): + cmd_output('git', 'init', str(tmpdir)) + tmpdir.join('.git/hooks').ensure_dir() + tmpdir.join('.git/hooks/pre-commit').write_binary( + b'#!/usr/bin/env bash\n' + b'# garbage: \xa0\xef\x12\xf2\n' + b'echo legacy hook\n', + ) + + with tmpdir.as_cwd(): + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 + + +FAIL_OLD_HOOK = re_assert.Matches( + r'fail!\n' + r'\[INFO\] Initializing environment for .+\.\n' + r'Bash hook\.+Passed\n', ) @@ -369,38 +405,42 @@ def test_failing_existing_hook_returns_1(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): # Write out a failing "old" hook - mkdirp(os.path.join(path, '.git/hooks')) - with io.open(os.path.join(path, '.git/hooks/pre-commit'), 'w') as f: + os.makedirs(os.path.join(path, '.git/hooks'), exist_ok=True) + with open(os.path.join(path, '.git/hooks/pre-commit'), 'w') as f: f.write('#!/usr/bin/env bash\necho "fail!"\nexit 1\n') make_executable(f.name) - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 # We should get a failure from the legacy hook ret, output = _get_commit_output(tempdir_factory) assert ret == 1 - assert FAIL_OLD_HOOK.match(output) + FAIL_OLD_HOOK.assert_matches(output) def test_install_overwrite_no_existing_hooks(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - assert install(C.CONFIG_FILE, store, overwrite=True) == 0 + assert not install( + C.CONFIG_FILE, store, hook_types=['pre-commit'], overwrite=True, + ) ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_install_overwrite(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): _write_legacy_hook(path) - assert install(C.CONFIG_FILE, store, overwrite=True) == 0 + assert not install( + C.CONFIG_FILE, store, hook_types=['pre-commit'], overwrite=True, + ) ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_uninstall_restores_legacy_hooks(tempdir_factory, store): @@ -409,13 +449,13 @@ def test_uninstall_restores_legacy_hooks(tempdir_factory, store): _write_legacy_hook(path) # Now install and uninstall pre-commit - assert install(C.CONFIG_FILE, store) == 0 - assert uninstall() == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 + assert uninstall(C.CONFIG_FILE, hook_types=['pre-commit']) == 0 # Make sure we installed the "old" hook correctly ret, output = _get_commit_output(tempdir_factory, touch_file='baz') assert ret == 0 - assert EXISTING_COMMIT_RUN.match(output) + EXISTING_COMMIT_RUN.assert_matches(output) def test_replace_old_commit_script(tempdir_factory, store): @@ -424,20 +464,20 @@ def test_replace_old_commit_script(tempdir_factory, store): # Install a script that looks like our old script pre_commit_contents = resource_text('hook-tmpl') new_contents = pre_commit_contents.replace( - CURRENT_HASH, PRIOR_HASHES[-1], + CURRENT_HASH.decode(), PRIOR_HASHES[-1].decode(), ) - mkdirp(os.path.join(path, '.git/hooks')) - with io.open(os.path.join(path, '.git/hooks/pre-commit'), 'w') as f: + os.makedirs(os.path.join(path, '.git/hooks'), exist_ok=True) + with open(os.path.join(path, '.git/hooks/pre-commit'), 'w') as f: f.write(new_contents) make_executable(f.name) # Install normally - assert install(C.CONFIG_FILE, store) == 0 + assert install(C.CONFIG_FILE, store, hook_types=['pre-commit']) == 0 ret, output = _get_commit_output(tempdir_factory) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) def test_uninstall_doesnt_remove_not_our_hooks(in_git_dir): @@ -445,48 +485,48 @@ def test_uninstall_doesnt_remove_not_our_hooks(in_git_dir): pre_commit.write('#!/usr/bin/env bash\necho 1\n') make_executable(pre_commit.strpath) - assert uninstall() == 0 + assert uninstall(C.CONFIG_FILE, hook_types=['pre-commit']) == 0 assert pre_commit.exists() -PRE_INSTALLED = re.compile( - r'Bash hook\.+Passed\r?\n' - r'\[master [a-f0-9]{7}\] commit!\r?\n' + - FILES_CHANGED + - r' create mode 100644 foo\r?\n$', +PRE_INSTALLED = re_assert.Matches( + fr'Bash hook\.+Passed\n' + fr'\[master [a-f0-9]{{7}}\] commit!\n' + fr'{FILES_CHANGED}' + fr' create mode 100644 foo\n$', ) def test_installs_hooks_with_hooks_True(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - install(C.CONFIG_FILE, store, hooks=True) + install(C.CONFIG_FILE, store, hook_types=['pre-commit'], hooks=True) ret, output = _get_commit_output( tempdir_factory, pre_commit_home=store.directory, ) assert ret == 0 - assert PRE_INSTALLED.match(output) + PRE_INSTALLED.assert_matches(output) def test_install_hooks_command(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) install_hooks(C.CONFIG_FILE, store) ret, output = _get_commit_output( tempdir_factory, pre_commit_home=store.directory, ) assert ret == 0 - assert PRE_INSTALLED.match(output) + PRE_INSTALLED.assert_matches(output) def test_installed_from_venv(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) # No environment so pre-commit is not on the path when running! # Should still pick up the python from when we installed ret, output = _get_commit_output( @@ -507,16 +547,14 @@ def test_installed_from_venv(tempdir_factory, store): }, ) assert ret == 0 - assert NORMAL_PRE_COMMIT_RUN.match(output) + NORMAL_PRE_COMMIT_RUN.assert_matches(output) -def _get_push_output(tempdir_factory, opts=()): +def _get_push_output(tempdir_factory, remote='origin', opts=()): return cmd_output_mocked_pre_commit_home( - 'git', 'push', 'origin', 'HEAD:new_branch', *opts, - # git push puts pre-commit to stderr - stderr=subprocess.STDOUT, + 'git', 'push', remote, 'HEAD:new_branch', *opts, tempdir_factory=tempdir_factory, - retcode=None + check=False, )[:2] @@ -525,7 +563,7 @@ def test_pre_push_integration_failing(tempdir_factory, store): path = tempdir_factory.get() cmd_output('git', 'clone', upstream, path) with cwd(path): - install(C.CONFIG_FILE, store, hook_type='pre-push') + install(C.CONFIG_FILE, store, hook_types=['pre-push']) # commit succeeds because pre-commit is only installed for pre-push assert _get_commit_output(tempdir_factory)[0] == 0 assert _get_commit_output(tempdir_factory, touch_file='zzz')[0] == 0 @@ -535,7 +573,7 @@ def test_pre_push_integration_failing(tempdir_factory, store): assert 'Failing hook' in output assert 'Failed' in output assert 'foo zzz' in output # both filenames should be printed - assert 'hookid: failing_hook' in output + assert 'hook id: failing_hook' in output def test_pre_push_integration_accepted(tempdir_factory, store): @@ -543,7 +581,7 @@ def test_pre_push_integration_accepted(tempdir_factory, store): path = tempdir_factory.get() cmd_output('git', 'clone', upstream, path) with cwd(path): - install(C.CONFIG_FILE, store, hook_type='pre-push') + install(C.CONFIG_FILE, store, hook_types=['pre-push']) assert _get_commit_output(tempdir_factory)[0] == 0 retc, output = _get_push_output(tempdir_factory) @@ -563,7 +601,7 @@ def test_pre_push_force_push_without_fetch(tempdir_factory, store): assert _get_push_output(tempdir_factory)[0] == 0 with cwd(path2): - install(C.CONFIG_FILE, store, hook_type='pre-push') + install(C.CONFIG_FILE, store, hook_types=['pre-push']) assert _get_commit_output(tempdir_factory, msg='force!')[0] == 0 retc, output = _get_push_output(tempdir_factory, opts=('--force',)) @@ -578,7 +616,7 @@ def test_pre_push_new_upstream(tempdir_factory, store): path = tempdir_factory.get() cmd_output('git', 'clone', upstream, path) with cwd(path): - install(C.CONFIG_FILE, store, hook_type='pre-push') + install(C.CONFIG_FILE, store, hook_types=['pre-push']) assert _get_commit_output(tempdir_factory)[0] == 0 cmd_output('git', 'remote', 'rename', 'origin', 'upstream') @@ -589,12 +627,39 @@ def test_pre_push_new_upstream(tempdir_factory, store): assert 'Passed' in output +def test_pre_push_environment_variables(tempdir_factory, store): + config = { + 'repo': 'local', + 'hooks': [ + { + 'id': 'print-remote-info', + 'name': 'print remote info', + 'entry': 'bash -c "echo remote: $PRE_COMMIT_REMOTE_NAME"', + 'language': 'system', + 'verbose': True, + }, + ], + } + + upstream = git_dir(tempdir_factory) + clone = tempdir_factory.get() + cmd_output('git', 'clone', upstream, clone) + add_config_to_repo(clone, config) + with cwd(clone): + install(C.CONFIG_FILE, store, hook_types=['pre-push']) + + cmd_output('git', 'remote', 'rename', 'origin', 'origin2') + retc, output = _get_push_output(tempdir_factory, remote='origin2') + assert retc == 0 + assert '\nremote: origin2\n' in output + + def test_pre_push_integration_empty_push(tempdir_factory, store): upstream = make_consuming_repo(tempdir_factory, 'script_hooks_repo') path = tempdir_factory.get() cmd_output('git', 'clone', upstream, path) with cwd(path): - install(C.CONFIG_FILE, store, hook_type='pre-push') + install(C.CONFIG_FILE, store, hook_types=['pre-push']) _get_push_output(tempdir_factory) retc, output = _get_push_output(tempdir_factory) assert output == 'Everything up-to-date\n' @@ -606,8 +671,8 @@ def test_pre_push_legacy(tempdir_factory, store): path = tempdir_factory.get() cmd_output('git', 'clone', upstream, path) with cwd(path): - mkdirp(os.path.join(path, '.git/hooks')) - with io.open(os.path.join(path, '.git/hooks/pre-push'), 'w') as f: + os.makedirs(os.path.join(path, '.git/hooks'), exist_ok=True) + with open(os.path.join(path, '.git/hooks/pre-push'), 'w') as f: f.write( '#!/usr/bin/env bash\n' 'set -eu\n' @@ -617,7 +682,7 @@ def test_pre_push_legacy(tempdir_factory, store): ) make_executable(f.name) - install(C.CONFIG_FILE, store, hook_type='pre-push') + install(C.CONFIG_FILE, store, hook_types=['pre-push']) assert _get_commit_output(tempdir_factory)[0] == 0 retc, output = _get_push_output(tempdir_factory) @@ -631,17 +696,20 @@ def test_pre_push_legacy(tempdir_factory, store): def test_commit_msg_integration_failing( commit_msg_repo, tempdir_factory, store, ): - install(C.CONFIG_FILE, store, hook_type='commit-msg') + install(C.CONFIG_FILE, store, hook_types=['commit-msg']) retc, out = _get_commit_output(tempdir_factory) assert retc == 1 - assert out.startswith('Must have "Signed off by:"...') - assert out.strip().endswith('...Failed') + assert out == '''\ +Must have "Signed off by:"...............................................Failed +- hook id: must-have-signoff +- exit code: 1 +''' def test_commit_msg_integration_passing( commit_msg_repo, tempdir_factory, store, ): - install(C.CONFIG_FILE, store, hook_type='commit-msg') + install(C.CONFIG_FILE, store, hook_types=['commit-msg']) msg = 'Hi\nSigned off by: me, lol' retc, out = _get_commit_output(tempdir_factory, msg=msg) assert retc == 0 @@ -652,8 +720,8 @@ def test_commit_msg_integration_passing( def test_commit_msg_legacy(commit_msg_repo, tempdir_factory, store): hook_path = os.path.join(commit_msg_repo, '.git/hooks/commit-msg') - mkdirp(os.path.dirname(hook_path)) - with io.open(hook_path, 'w') as hook_file: + os.makedirs(os.path.dirname(hook_path), exist_ok=True) + with open(hook_path, 'w') as hook_file: hook_file.write( '#!/usr/bin/env bash\n' 'set -eu\n' @@ -662,7 +730,7 @@ def test_commit_msg_legacy(commit_msg_repo, tempdir_factory, store): ) make_executable(hook_path) - install(C.CONFIG_FILE, store, hook_type='commit-msg') + install(C.CONFIG_FILE, store, hook_types=['commit-msg']) msg = 'Hi\nSigned off by: asottile' retc, out = _get_commit_output(tempdir_factory, msg=msg) @@ -672,22 +740,237 @@ def test_commit_msg_legacy(commit_msg_repo, tempdir_factory, store): assert second_line.startswith('Must have "Signed off by:"...') +def test_post_commit_integration(tempdir_factory, store): + path = git_dir(tempdir_factory) + config = { + 'repos': [ + { + 'repo': 'local', + 'hooks': [{ + 'id': 'post-commit', + 'name': 'Post commit', + 'entry': 'touch post-commit.tmp', + 'language': 'system', + 'always_run': True, + 'verbose': True, + 'stages': ['post-commit'], + }], + }, + ], + } + write_config(path, config) + with cwd(path): + _get_commit_output(tempdir_factory) + assert not os.path.exists('post-commit.tmp') + + install(C.CONFIG_FILE, store, hook_types=['post-commit']) + _get_commit_output(tempdir_factory) + assert os.path.exists('post-commit.tmp') + + +def test_post_merge_integration(tempdir_factory, store): + path = git_dir(tempdir_factory) + config = { + 'repos': [ + { + 'repo': 'local', + 'hooks': [{ + 'id': 'post-merge', + 'name': 'Post merge', + 'entry': 'touch post-merge.tmp', + 'language': 'system', + 'always_run': True, + 'verbose': True, + 'stages': ['post-merge'], + }], + }, + ], + } + write_config(path, config) + with cwd(path): + # create a simple diamond of commits for a non-trivial merge + open('init', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + + open('master', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + + cmd_output('git', 'checkout', '-b', 'branch', 'HEAD^') + open('branch', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + + cmd_output('git', 'checkout', 'master') + install(C.CONFIG_FILE, store, hook_types=['post-merge']) + retc, stdout, stderr = cmd_output_mocked_pre_commit_home( + 'git', 'merge', 'branch', + tempdir_factory=tempdir_factory, + ) + assert retc == 0 + assert os.path.exists('post-merge.tmp') + + +def test_pre_rebase_integration(tempdir_factory, store): + path = git_dir(tempdir_factory) + config = { + 'repos': [ + { + 'repo': 'local', + 'hooks': [{ + 'id': 'pre-rebase', + 'name': 'Pre rebase', + 'entry': 'touch pre-rebase.tmp', + 'language': 'system', + 'always_run': True, + 'verbose': True, + 'stages': ['pre-rebase'], + }], + }, + ], + } + write_config(path, config) + with cwd(path): + install(C.CONFIG_FILE, store, hook_types=['pre-rebase']) + open('foo', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + + cmd_output('git', 'checkout', '-b', 'branch') + open('bar', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + + cmd_output('git', 'checkout', 'master') + open('baz', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + + cmd_output('git', 'checkout', 'branch') + cmd_output('git', 'rebase', 'master', 'branch') + assert os.path.exists('pre-rebase.tmp') + + +def test_post_rewrite_integration(tempdir_factory, store): + path = git_dir(tempdir_factory) + config = { + 'repos': [ + { + 'repo': 'local', + 'hooks': [{ + 'id': 'post-rewrite', + 'name': 'Post rewrite', + 'entry': 'touch post-rewrite.tmp', + 'language': 'system', + 'always_run': True, + 'verbose': True, + 'stages': ['post-rewrite'], + }], + }, + ], + } + write_config(path, config) + with cwd(path): + open('init', 'a').close() + cmd_output('git', 'add', '.') + install(C.CONFIG_FILE, store, hook_types=['post-rewrite']) + git_commit() + + assert not os.path.exists('post-rewrite.tmp') + + git_commit('--amend', '-m', 'ammended message') + assert os.path.exists('post-rewrite.tmp') + + +def test_post_checkout_integration(tempdir_factory, store): + path = git_dir(tempdir_factory) + config = { + 'repos': [ + { + 'repo': 'local', + 'hooks': [{ + 'id': 'post-checkout', + 'name': 'Post checkout', + 'entry': 'bash -c "echo ${PRE_COMMIT_TO_REF}"', + 'language': 'system', + 'always_run': True, + 'verbose': True, + 'stages': ['post-checkout'], + }], + }, + {'repo': 'meta', 'hooks': [{'id': 'identity'}]}, + ], + } + write_config(path, config) + with cwd(path): + cmd_output('git', 'add', '.') + git_commit() + + # add a file only on `feature`, it should not be passed to hooks + cmd_output('git', 'checkout', '-b', 'feature') + open('some_file', 'a').close() + cmd_output('git', 'add', '.') + git_commit() + cmd_output('git', 'checkout', 'master') + + install(C.CONFIG_FILE, store, hook_types=['post-checkout']) + retc, _, stderr = cmd_output('git', 'checkout', 'feature') + assert stderr is not None + assert retc == 0 + assert git.head_rev(path) in stderr + assert 'some_file' not in stderr + + +def test_skips_post_checkout_unstaged_changes(tempdir_factory, store): + path = git_dir(tempdir_factory) + config = { + 'repo': 'local', + 'hooks': [{ + 'id': 'fail', + 'name': 'fail', + 'entry': 'fail', + 'language': 'fail', + 'always_run': True, + 'stages': ['post-checkout'], + }], + } + write_config(path, config) + with cwd(path): + cmd_output('git', 'add', '.') + _get_commit_output(tempdir_factory) + + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) + install(C.CONFIG_FILE, store, hook_types=['post-checkout']) + + # make an unstaged change so staged_files_only fires + open('file', 'a').close() + cmd_output('git', 'add', 'file') + with open('file', 'w') as f: + f.write('unstaged changes') + + retc, out = _get_commit_output(tempdir_factory, all_files=False) + assert retc == 0 + + def test_prepare_commit_msg_integration_failing( failing_prepare_commit_msg_repo, tempdir_factory, store, ): - install(C.CONFIG_FILE, store, hook_type='prepare-commit-msg') + install(C.CONFIG_FILE, store, hook_types=['prepare-commit-msg']) retc, out = _get_commit_output(tempdir_factory) assert retc == 1 - assert out.startswith('Add "Signed off by:"...') - assert out.strip().endswith('...Failed') + assert out == '''\ +Add "Signed off by:".....................................................Failed +- hook id: add-signoff +- exit code: 1 +''' def test_prepare_commit_msg_integration_passing( prepare_commit_msg_repo, tempdir_factory, store, ): - install(C.CONFIG_FILE, store, hook_type='prepare-commit-msg') - msg = 'Hi' - retc, out = _get_commit_output(tempdir_factory, msg=msg) + install(C.CONFIG_FILE, store, hook_types=['prepare-commit-msg']) + retc, out = _get_commit_output(tempdir_factory, msg='Hi') assert retc == 0 first_line = out.splitlines()[0] assert first_line.startswith('Add "Signed off by:"...') @@ -695,7 +978,7 @@ def test_prepare_commit_msg_integration_passing( commit_msg_path = os.path.join( prepare_commit_msg_repo, '.git/COMMIT_EDITMSG', ) - with io.open(commit_msg_path) as f: + with open(commit_msg_path) as f: assert 'Signed off by: ' in f.read() @@ -705,8 +988,8 @@ def test_prepare_commit_msg_legacy( hook_path = os.path.join( prepare_commit_msg_repo, '.git/hooks/prepare-commit-msg', ) - mkdirp(os.path.dirname(hook_path)) - with io.open(hook_path, 'w') as hook_file: + os.makedirs(os.path.dirname(hook_path), exist_ok=True) + with open(hook_path, 'w') as hook_file: hook_file.write( '#!/usr/bin/env bash\n' 'set -eu\n' @@ -715,10 +998,9 @@ def test_prepare_commit_msg_legacy( ) make_executable(hook_path) - install(C.CONFIG_FILE, store, hook_type='prepare-commit-msg') + install(C.CONFIG_FILE, store, hook_types=['prepare-commit-msg']) - msg = 'Hi' - retc, out = _get_commit_output(tempdir_factory, msg=msg) + retc, out = _get_commit_output(tempdir_factory, msg='Hi') assert retc == 0 first_line, second_line = out.splitlines()[:2] assert first_line == 'legacy' @@ -726,16 +1008,43 @@ def test_prepare_commit_msg_legacy( commit_msg_path = os.path.join( prepare_commit_msg_repo, '.git/COMMIT_EDITMSG', ) - with io.open(commit_msg_path) as f: + with open(commit_msg_path) as f: assert 'Signed off by: ' in f.read() +def test_pre_merge_commit_integration(tempdir_factory, store): + output_pattern = re_assert.Matches( + r'^\[INFO\] Initializing environment for .+\n' + r'Bash hook\.+Passed\n' + r"Merge made by the '(ort|recursive)' strategy.\n" + r' foo \| 0\n' + r' 1 file changed, 0 insertions\(\+\), 0 deletions\(-\)\n' + r' create mode 100644 foo\n$', + ) + + path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') + with cwd(path): + ret = install(C.CONFIG_FILE, store, hook_types=['pre-merge-commit']) + assert ret == 0 + + cmd_output('git', 'checkout', 'master', '-b', 'feature') + _get_commit_output(tempdir_factory) + cmd_output('git', 'checkout', 'master') + ret, output, _ = cmd_output_mocked_pre_commit_home( + 'git', 'merge', '--no-ff', '--no-edit', 'feature', + tempdir_factory=tempdir_factory, + ) + assert ret == 0 + output_pattern.assert_matches(output) + + def test_install_disallow_missing_config(tempdir_factory, store): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') with cwd(path): remove_config_from_repo(path) ret = install( - C.CONFIG_FILE, store, overwrite=True, skip_on_missing_config=False, + C.CONFIG_FILE, store, hook_types=['pre-commit'], + overwrite=True, skip_on_missing_config=False, ) assert ret == 0 @@ -748,7 +1057,8 @@ def test_install_allow_missing_config(tempdir_factory, store): with cwd(path): remove_config_from_repo(path) ret = install( - C.CONFIG_FILE, store, overwrite=True, skip_on_missing_config=True, + C.CONFIG_FILE, store, hook_types=['pre-commit'], + overwrite=True, skip_on_missing_config=True, ) assert ret == 0 @@ -766,7 +1076,8 @@ def test_install_temporarily_allow_mising_config(tempdir_factory, store): with cwd(path): remove_config_from_repo(path) ret = install( - C.CONFIG_FILE, store, overwrite=True, skip_on_missing_config=False, + C.CONFIG_FILE, store, hook_types=['pre-commit'], + overwrite=True, skip_on_missing_config=False, ) assert ret == 0 @@ -778,3 +1089,16 @@ def test_install_temporarily_allow_mising_config(tempdir_factory, store): 'Skipping `pre-commit`.' ) assert expected in output + + +def test_install_uninstall_default_hook_types(in_git_dir, store): + cfg_src = 'default_install_hook_types: [pre-commit, pre-push]\nrepos: []\n' + in_git_dir.join(C.CONFIG_FILE).write(cfg_src) + + assert not install(C.CONFIG_FILE, store, hook_types=None) + assert os.access(in_git_dir.join('.git/hooks/pre-commit').strpath, os.X_OK) + assert os.access(in_git_dir.join('.git/hooks/pre-push').strpath, os.X_OK) + + assert not uninstall(C.CONFIG_FILE, hook_types=None) + assert not in_git_dir.join('.git/hooks/pre-commit').exists() + assert not in_git_dir.join('.git/hooks/pre-push').exists() diff --git a/tests/commands/migrate_config_test.py b/tests/commands/migrate_config_test.py index c58b9f74..a517d2f4 100644 --- a/tests/commands/migrate_config_test.py +++ b/tests/commands/migrate_config_test.py @@ -1,25 +1,26 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations + +from unittest import mock import pytest +import yaml import pre_commit.constants as C -from pre_commit.commands.migrate_config import _indent +from pre_commit.clientlib import InvalidConfigError from pre_commit.commands.migrate_config import migrate_config +from pre_commit.yaml import yaml_compose -@pytest.mark.parametrize( - ('s', 'expected'), - ( - ('', ''), - ('a', ' a'), - ('foo\nbar', ' foo\n bar'), - ('foo\n\nbar\n', ' foo\n\n bar\n'), - ('\n\n\n', '\n\n\n'), - ), -) -def test_indent(s, expected): - assert _indent(s) == expected +@pytest.fixture(autouse=True, params=['c', 'pure']) +def switch_pyyaml_impl(request): + if request.param == 'c': + yield + else: + with mock.patch.dict( + yaml_compose.keywords, + {'Loader': yaml.SafeLoader}, + ): + yield def test_migrate_config_normal_format(tmpdir, capsys): @@ -149,11 +150,132 @@ def test_migrate_config_sha_to_rev(tmpdir): ) -@pytest.mark.parametrize('contents', ('', '\n')) -def test_empty_configuration_file_user_error(tmpdir, contents): +def test_migrate_config_sha_to_rev_json(tmp_path): + contents = """\ +{"repos": [{ + "repo": "https://github.com/pre-commit/pre-commit-hooks", + "sha": "v1.2.0", + "hooks": [] +}]} +""" + expected = """\ +{"repos": [{ + "repo": "https://github.com/pre-commit/pre-commit-hooks", + "rev": "v1.2.0", + "hooks": [] +}]} +""" + cfg = tmp_path.joinpath('cfg.yaml') + cfg.write_text(contents) + assert not migrate_config(str(cfg)) + assert cfg.read_text() == expected + + +def test_migrate_config_language_python_venv(tmp_path): + src = '''\ +repos: +- repo: local + hooks: + - id: example + name: example + entry: example + language: python_venv + - id: example + name: example + entry: example + language: system +''' + expected = '''\ +repos: +- repo: local + hooks: + - id: example + name: example + entry: example + language: python + - id: example + name: example + entry: example + language: system +''' + cfg = tmp_path.joinpath('cfg.yaml') + cfg.write_text(src) + assert migrate_config(str(cfg)) == 0 + assert cfg.read_text() == expected + + +def test_migrate_config_quoted_python_venv(tmp_path): + src = '''\ +repos: +- repo: local + hooks: + - id: example + name: example + entry: example + language: "python_venv" +''' + expected = '''\ +repos: +- repo: local + hooks: + - id: example + name: example + entry: example + language: "python" +''' + cfg = tmp_path.joinpath('cfg.yaml') + cfg.write_text(src) + assert migrate_config(str(cfg)) == 0 + assert cfg.read_text() == expected + + +def test_migrate_config_default_stages(tmp_path): + src = '''\ +default_stages: [commit, push, merge-commit, commit-msg] +repos: [] +''' + expected = '''\ +default_stages: [pre-commit, pre-push, pre-merge-commit, commit-msg] +repos: [] +''' + cfg = tmp_path.joinpath('cfg.yaml') + cfg.write_text(src) + assert migrate_config(str(cfg)) == 0 + assert cfg.read_text() == expected + + +def test_migrate_config_hook_stages(tmp_path): + src = '''\ +repos: +- repo: local + hooks: + - id: example + name: example + entry: example + language: system + stages: ["commit", "push", "merge-commit", "commit-msg"] +''' + expected = '''\ +repos: +- repo: local + hooks: + - id: example + name: example + entry: example + language: system + stages: ["pre-commit", "pre-push", "pre-merge-commit", "commit-msg"] +''' + cfg = tmp_path.joinpath('cfg.yaml') + cfg.write_text(src) + assert migrate_config(str(cfg)) == 0 + assert cfg.read_text() == expected + + +def test_migrate_config_invalid_yaml(tmpdir): + contents = '[' cfg = tmpdir.join(C.CONFIG_FILE) cfg.write(contents) - with tmpdir.as_cwd(): - assert not migrate_config(C.CONFIG_FILE) - # even though the config is invalid, this should be a noop - assert cfg.read() == contents + with tmpdir.as_cwd(), pytest.raises(InvalidConfigError) as excinfo: + migrate_config(C.CONFIG_FILE) + expected = '\n==> File .pre-commit-config.yaml\n=====> ' + assert str(excinfo.value).startswith(expected) diff --git a/tests/commands/run_test.py b/tests/commands/run_test.py index 94d44e15..e4af1e16 100644 --- a/tests/commands/run_test.py +++ b/tests/commands/run_test.py @@ -1,18 +1,22 @@ -# -*- coding: UTF-8 -*- -from __future__ import unicode_literals +from __future__ import annotations -import io import os.path -import subprocess +import shlex import sys +import time +from collections.abc import MutableMapping +from unittest import mock import pytest import pre_commit.constants as C +from pre_commit import color from pre_commit.commands.install_uninstall import install from pre_commit.commands.run import _compute_cols +from pre_commit.commands.run import _full_msg from pre_commit.commands.run import _get_skips from pre_commit.commands.run import _has_unmerged_paths +from pre_commit.commands.run import _start_msg from pre_commit.commands.run import Classifier from pre_commit.commands.run import filter_by_include_exclude from pre_commit.commands.run import run @@ -20,15 +24,84 @@ from pre_commit.util import cmd_output from pre_commit.util import make_executable from testing.auto_namedtuple import auto_namedtuple from testing.fixtures import add_config_to_repo +from testing.fixtures import git_dir from testing.fixtures import make_consuming_repo from testing.fixtures import modify_config from testing.fixtures import read_config from testing.fixtures import sample_meta_config +from testing.fixtures import write_config from testing.util import cmd_output_mocked_pre_commit_home from testing.util import cwd from testing.util import git_commit from testing.util import run_opts -from testing.util import xfailif_no_symlink + + +def test_start_msg(): + ret = _start_msg(start='start', end_len=5, cols=15) + # 4 dots: 15 - 5 - 5 - 1 + assert ret == 'start....' + + +def test_full_msg(): + ret = _full_msg( + start='start', + end_msg='end', + end_color='', + use_color=False, + cols=15, + ) + # 6 dots: 15 - 5 - 3 - 1 + assert ret == 'start......end\n' + + +def test_full_msg_with_cjk(): + ret = _full_msg( + start='ε•Šγ‚μ•„', + end_msg='end', + end_color='', + use_color=False, + cols=15, + ) + # 5 dots: 15 - 6 - 3 - 1 + assert ret == 'ε•Šγ‚μ•„.....end\n' + + +def test_full_msg_with_color(): + ret = _full_msg( + start='start', + end_msg='end', + end_color=color.RED, + use_color=True, + cols=15, + ) + # 6 dots: 15 - 5 - 3 - 1 + assert ret == f'start......{color.RED}end{color.NORMAL}\n' + + +def test_full_msg_with_postfix(): + ret = _full_msg( + start='start', + postfix='post ', + end_msg='end', + end_color='', + use_color=False, + cols=20, + ) + # 6 dots: 20 - 5 - 5 - 3 - 1 + assert ret == 'start......post end\n' + + +def test_full_msg_postfix_not_colored(): + ret = _full_msg( + start='start', + postfix='post ', + end_msg='end', + end_color=color.RED, + use_color=True, + cols=20, + ) + # 6 dots: 20 - 5 - 5 - 3 - 1 + assert ret == f'start......post {color.RED}end{color.NORMAL}\n' @pytest.fixture @@ -92,7 +165,7 @@ def test_run_all_hooks_failing(cap_out, store, repo_with_failing_hook): ( b'Failing hook', b'Failed', - b'hookid: failing_hook', + b'hook id: failing_hook', b'Fail\nfoo.py\n', ), expected_ret=1, @@ -123,14 +196,14 @@ def test_hook_that_modifies_but_returns_zero(cap_out, store, tempdir_factory): # The first should fail b'Failed', # With a modified file (default message + the hook's output) - b'Files were modified by this hook. Additional output:\n\n' + b'- files were modified by this hook\n\n' b'Modified: foo.py', # The next hook should pass despite the first modifying b'Passed', # The next hook should fail b'Failed', # bar.py was modified, but provides no additional output - b'Files were modified by this hook.\n', + b'- files were modified by this hook\n', ), 1, True, @@ -148,10 +221,23 @@ def test_types_hook_repository(cap_out, store, tempdir_factory): assert b'bar.notpy' not in printed +def test_types_or_hook_repository(cap_out, store, tempdir_factory): + git_path = make_consuming_repo(tempdir_factory, 'types_or_repo') + with cwd(git_path): + stage_a_file('bar.notpy') + stage_a_file('bar.pxd') + stage_a_file('bar.py') + ret, printed = _do_run(cap_out, store, git_path, run_opts()) + assert ret == 1 + assert b'bar.notpy' not in printed + assert b'bar.pxd' in printed + assert b'bar.py' in printed + + def test_exclude_types_hook_repository(cap_out, store, tempdir_factory): git_path = make_consuming_repo(tempdir_factory, 'exclude_types_repo') with cwd(git_path): - with io.open('exe', 'w') as exe: + with open('exe', 'w') as exe: exe.write('#!/usr/bin/env python3\n') make_executable('exe') cmd_output('git', 'add', 'exe') @@ -162,20 +248,55 @@ def test_exclude_types_hook_repository(cap_out, store, tempdir_factory): assert b'exe' not in printed -def test_global_exclude(cap_out, store, tempdir_factory): - git_path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') - with cwd(git_path): - with modify_config() as config: - config['exclude'] = '^foo.py$' - open('foo.py', 'a').close() - open('bar.py', 'a').close() - cmd_output('git', 'add', '.') - opts = run_opts(verbose=True) - ret, printed = _do_run(cap_out, store, git_path, opts) - assert ret == 0 - # Does not contain foo.py since it was excluded - expected = b'hookid: bash_hook\n\nbar.py\nHello World\n\n' - assert printed.endswith(expected) +def test_global_exclude(cap_out, store, in_git_dir): + config = { + 'exclude': r'^foo\.py$', + 'repos': [{'repo': 'meta', 'hooks': [{'id': 'identity'}]}], + } + write_config('.', config) + open('foo.py', 'a').close() + open('bar.py', 'a').close() + cmd_output('git', 'add', '.') + opts = run_opts(verbose=True) + ret, printed = _do_run(cap_out, store, str(in_git_dir), opts) + assert ret == 0 + # Does not contain foo.py since it was excluded + assert printed.startswith(f'identity{"." * 65}Passed\n'.encode()) + assert printed.endswith(b'\n\n.pre-commit-config.yaml\nbar.py\n\n') + + +def test_global_files(cap_out, store, in_git_dir): + config = { + 'files': r'^bar\.py$', + 'repos': [{'repo': 'meta', 'hooks': [{'id': 'identity'}]}], + } + write_config('.', config) + open('foo.py', 'a').close() + open('bar.py', 'a').close() + cmd_output('git', 'add', '.') + opts = run_opts(verbose=True) + ret, printed = _do_run(cap_out, store, str(in_git_dir), opts) + assert ret == 0 + # Does not contain foo.py since it was excluded + assert printed.startswith(f'identity{"." * 65}Passed\n'.encode()) + assert printed.endswith(b'\n\nbar.py\n\n') + + +@pytest.mark.parametrize( + ('t1', 't2', 'expected'), + ( + (1.234, 2., b'\n- duration: 0.77s\n'), + (1., 1., b'\n- duration: 0s\n'), + ), +) +def test_verbose_duration(cap_out, store, in_git_dir, t1, t2, expected): + write_config('.', {'repo': 'meta', 'hooks': [{'id': 'identity'}]}) + cmd_output('git', 'add', '.') + opts = run_opts(verbose=True) + with mock.patch.object(time, 'monotonic', side_effect=(t1, t2)): + ret, printed = _do_run(cap_out, store, str(in_git_dir), opts) + assert ret == 0 + assert expected in printed @pytest.mark.parametrize( @@ -233,13 +354,13 @@ def test_show_diff_on_failure( ({'hook': 'bash_hook'}, (b'Bash hook', b'Passed'), 0, True), ( {'hook': 'nope'}, - (b'No hook with id `nope` in stage `commit`',), + (b'No hook with id `nope` in stage `pre-commit`',), 1, True, ), ( - {'hook': 'nope', 'hook_stage': 'push'}, - (b'No hook with id `nope` in stage `push`',), + {'hook': 'nope', 'hook_stage': 'pre-push'}, + (b'No hook with id `nope` in stage `pre-push`',), 1, True, ), @@ -353,22 +474,54 @@ def test_hook_verbose_enabled(cap_out, store, repo_with_passing_hook): @pytest.mark.parametrize( - ('origin', 'source'), (('master', ''), ('', 'master')), + ('from_ref', 'to_ref'), (('master', ''), ('', 'master')), ) -def test_origin_source_error_msg_error( - cap_out, store, repo_with_passing_hook, origin, source, +def test_from_ref_to_ref_error_msg_error( + cap_out, store, repo_with_passing_hook, from_ref, to_ref, ): - args = run_opts(origin=origin, source=source) + args = run_opts(from_ref=from_ref, to_ref=to_ref) ret, printed = _do_run(cap_out, store, repo_with_passing_hook, args) assert ret == 1 - assert b'Specify both --origin and --source.' in printed + assert b'Specify both --from-ref and --to-ref.' in printed -def test_origin_source_both_ok(cap_out, store, repo_with_passing_hook): - args = run_opts(origin='master', source='master') +def test_all_push_options_ok(cap_out, store, repo_with_passing_hook): + args = run_opts( + from_ref='master', to_ref='master', + remote_branch='master', + local_branch='master', + remote_name='origin', remote_url='https://example.com/repo', + ) ret, printed = _do_run(cap_out, store, repo_with_passing_hook, args) assert ret == 0 - assert b'Specify both --origin and --source.' not in printed + assert b'Specify both --from-ref and --to-ref.' not in printed + + +def test_is_squash_merge(cap_out, store, repo_with_passing_hook): + args = run_opts(is_squash_merge='1') + environ: MutableMapping[str, str] = {} + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, args, environ, + ) + assert environ['PRE_COMMIT_IS_SQUASH_MERGE'] == '1' + + +def test_rewrite_command(cap_out, store, repo_with_passing_hook): + args = run_opts(rewrite_command='amend') + environ: MutableMapping[str, str] = {} + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, args, environ, + ) + assert environ['PRE_COMMIT_REWRITE_COMMAND'] == 'amend' + + +def test_checkout_type(cap_out, store, repo_with_passing_hook): + args = run_opts(from_ref='', to_ref='', checkout_type='1') + environ: MutableMapping[str, str] = {} + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, args, environ, + ) + assert environ['PRE_COMMIT_CHECKOUT_TYPE'] == '1' def test_has_unmerged_paths(in_merge_conflict): @@ -383,11 +536,18 @@ def test_merge_conflict(cap_out, store, in_merge_conflict): assert b'Unmerged files. Resolve before committing.' in printed +def test_files_during_merge_conflict(cap_out, store, in_merge_conflict): + opts = run_opts(files=['placeholder']) + ret, printed = _do_run(cap_out, store, in_merge_conflict, opts) + assert ret == 0 + assert b'Bash hook' in printed + + def test_merge_conflict_modified(cap_out, store, in_merge_conflict): # Touch another file so we have unstaged non-conflicting things - assert os.path.exists('dummy') - with open('dummy', 'w') as dummy_file: - dummy_file.write('bar\nbaz\n') + assert os.path.exists('placeholder') + with open('placeholder', 'w') as placeholder_file: + placeholder_file.write('bar\nbaz\n') ret, printed = _do_run(cap_out, store, in_merge_conflict, run_opts()) assert ret == 1 @@ -403,24 +563,32 @@ def test_merge_conflict_resolved(cap_out, store, in_merge_conflict): assert msg in printed +def test_rebase(cap_out, store, repo_with_passing_hook): + args = run_opts(pre_rebase_upstream='master', pre_rebase_branch='topic') + environ: MutableMapping[str, str] = {} + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, args, environ, + ) + assert environ['PRE_COMMIT_PRE_REBASE_UPSTREAM'] == 'master' + assert environ['PRE_COMMIT_PRE_REBASE_BRANCH'] == 'topic' + + @pytest.mark.parametrize( - ('hooks', 'verbose', 'expected'), + ('hooks', 'expected'), ( - ([], True, 80), - ([auto_namedtuple(id='a', name='a' * 51)], False, 81), - ([auto_namedtuple(id='a', name='a' * 51)], True, 85), + ([], 80), + ([auto_namedtuple(id='a', name='a' * 51)], 81), ( [ auto_namedtuple(id='a', name='a' * 51), auto_namedtuple(id='b', name='b' * 52), ], - False, 82, ), ), ) -def test_compute_cols(hooks, verbose, expected): - assert _compute_cols(hooks, verbose) == expected +def test_compute_cols(hooks, expected): + assert _compute_cols(hooks) == expected @pytest.mark.parametrize( @@ -461,6 +629,55 @@ def test_skip_aliased_hook(cap_out, store, aliased_repo): assert printed.count(msg) == 1 +def test_skip_bypasses_installation(cap_out, store, repo_with_passing_hook): + config = { + 'repo': 'local', + 'hooks': [ + { + 'id': 'skipme', + 'name': 'skipme', + 'entry': 'skipme', + 'language': 'python', + 'additional_dependencies': ['/pre-commit-does-not-exist'], + }, + ], + } + add_config_to_repo(repo_with_passing_hook, config) + + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, + run_opts(all_files=True), + {'SKIP': 'skipme'}, + ) + assert ret == 0 + + +def test_skip_alias_bypasses_installation( + cap_out, store, repo_with_passing_hook, +): + config = { + 'repo': 'local', + 'hooks': [ + { + 'id': 'skipme', + 'name': 'skipme-1', + 'alias': 'skipme-1', + 'entry': 'skipme', + 'language': 'python', + 'additional_dependencies': ['/pre-commit-does-not-exist'], + }, + ], + } + add_config_to_repo(repo_with_passing_hook, config) + + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, + run_opts(all_files=True), + {'SKIP': 'skipme-1'}, + ) + assert ret == 0 + + def test_hook_id_not_in_non_verbose_output( cap_out, store, repo_with_passing_hook, ): @@ -474,7 +691,7 @@ def test_hook_id_in_verbose_output(cap_out, store, repo_with_passing_hook): ret, printed = _do_run( cap_out, store, repo_with_passing_hook, run_opts(verbose=True), ) - assert b'[bash_hook] Bash hook' in printed + assert b'- hook id: bash_hook' in printed def test_multiple_hooks_same_id(cap_out, store, repo_with_passing_hook): @@ -511,7 +728,7 @@ def test_non_ascii_hook_id(repo_with_passing_hook, tempdir_factory): with cwd(repo_with_passing_hook): _, stdout, _ = cmd_output_mocked_pre_commit_home( sys.executable, '-m', 'pre_commit.main', 'run', 'β˜ƒ', - retcode=None, tempdir_factory=tempdir_factory, + check=False, tempdir_factory=tempdir_factory, ) assert 'UnicodeDecodeError' not in stdout # Doesn't actually happen, but a reasonable assertion @@ -524,19 +741,17 @@ def test_stdout_write_bug_py26(repo_with_failing_hook, store, tempdir_factory): config['repos'][0]['hooks'][0]['args'] = ['β˜ƒ'] stage_a_file() - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) # Have to use subprocess because pytest monkeypatches sys.stdout - _, stdout, _ = git_commit( + _, out = git_commit( fn=cmd_output_mocked_pre_commit_home, - # git commit puts pre-commit to stderr - stderr=subprocess.STDOUT, - retcode=None, tempdir_factory=tempdir_factory, + check=False, ) - assert 'UnicodeEncodeError' not in stdout + assert 'UnicodeEncodeError' not in out # Doesn't actually happen, but a reasonable assertion - assert 'UnicodeDecodeError' not in stdout + assert 'UnicodeDecodeError' not in out def test_lots_of_files(store, tempdir_factory): @@ -550,37 +765,70 @@ def test_lots_of_files(store, tempdir_factory): # Write a crap ton of files for i in range(400): - filename = '{}{}'.format('a' * 100, i) - open(filename, 'w').close() + open(f'{"a" * 100}{i}', 'w').close() cmd_output('git', 'add', '.') - install(C.CONFIG_FILE, store) + install(C.CONFIG_FILE, store, hook_types=['pre-commit']) git_commit( fn=cmd_output_mocked_pre_commit_home, - # git commit puts pre-commit to stderr - stderr=subprocess.STDOUT, tempdir_factory=tempdir_factory, ) +def test_no_textconv(cap_out, store, repo_with_passing_hook): + # git textconv filters can hide changes from hooks + with open('.gitattributes', 'w') as fp: + fp.write('*.jpeg diff=empty\n') + + with open('.git/config', 'a') as fp: + fp.write('[diff "empty"]\n') + fp.write('textconv = "true"\n') + + config = { + 'repo': 'local', + 'hooks': [ + { + 'id': 'extend-jpeg', + 'name': 'extend-jpeg', + 'language': 'system', + 'entry': ( + f'{shlex.quote(sys.executable)} -c "import sys; ' + 'open(sys.argv[1], \'ab\').write(b\'\\x00\')"' + ), + 'types': ['jpeg'], + }, + ], + } + add_config_to_repo(repo_with_passing_hook, config) + + stage_a_file('example.jpeg') + + _test_run( + cap_out, + store, + repo_with_passing_hook, + {}, + ( + b'Failed', + ), + expected_ret=1, + stage=False, + ) + + def test_stages(cap_out, store, repo_with_passing_hook): config = { 'repo': 'local', 'hooks': [ { - 'id': 'do-not-commit-{}'.format(i), - 'name': 'hook {}'.format(i), + 'id': f'do-not-commit-{i}', + 'name': f'hook {i}', 'entry': 'DO NOT COMMIT', 'language': 'pygrep', 'stages': [stage], } - for i, stage in enumerate( - ( - 'commit', 'push', 'manual', 'prepare-commit-msg', - 'commit-msg', - ), 1, - ) + for i, stage in enumerate(('pre-commit', 'pre-push', 'manual'), 1) ], } add_config_to_repo(repo_with_passing_hook, config) @@ -595,16 +843,14 @@ def test_stages(cap_out, store, repo_with_passing_hook): assert printed.count(b'hook ') == 1 return printed - assert _run_for_stage('commit').startswith(b'hook 1...') - assert _run_for_stage('push').startswith(b'hook 2...') + assert _run_for_stage('pre-commit').startswith(b'hook 1...') + assert _run_for_stage('pre-push').startswith(b'hook 2...') assert _run_for_stage('manual').startswith(b'hook 3...') - assert _run_for_stage('prepare-commit-msg').startswith(b'hook 4...') - assert _run_for_stage('commit-msg').startswith(b'hook 5...') def test_commit_msg_hook(cap_out, store, commit_msg_repo): filename = '.git/COMMIT_EDITMSG' - with io.open(filename, 'w') as f: + with open(filename, 'w') as f: f.write('This is the commit message') _test_run( @@ -618,22 +864,48 @@ def test_commit_msg_hook(cap_out, store, commit_msg_repo): ) +def test_post_checkout_hook(cap_out, store, tempdir_factory): + path = git_dir(tempdir_factory) + config = { + 'repo': 'meta', 'hooks': [ + {'id': 'identity', 'stages': ['post-checkout']}, + ], + } + add_config_to_repo(path, config) + + with cwd(path): + _test_run( + cap_out, + store, + path, + {'hook_stage': 'post-checkout'}, + expected_outputs=[b'identity...'], + expected_ret=0, + stage=False, + ) + + def test_prepare_commit_msg_hook(cap_out, store, prepare_commit_msg_repo): filename = '.git/COMMIT_EDITMSG' - with io.open(filename, 'w') as f: + with open(filename, 'w') as f: f.write('This is the commit message') _test_run( cap_out, store, prepare_commit_msg_repo, - {'hook_stage': 'prepare-commit-msg', 'commit_msg_filename': filename}, + { + 'hook_stage': 'prepare-commit-msg', + 'commit_msg_filename': filename, + 'prepare_commit_message_source': 'commit', + 'commit_object_name': 'HEAD', + }, expected_outputs=[b'Add "Signed off by:"', b'Passed'], expected_ret=0, stage=False, ) - with io.open(filename) as f: + with open(filename) as f: assert 'Signed off by: ' in f.read() @@ -642,9 +914,11 @@ def test_local_hook_passes(cap_out, store, repo_with_passing_hook): 'repo': 'local', 'hooks': [ { - 'id': 'flake8', - 'name': 'flake8', - 'entry': "'{}' -m flake8".format(sys.executable), + 'id': 'identity-copy', + 'name': 'identity-copy', + 'entry': '{} -m pre_commit.meta_hooks.identity'.format( + shlex.quote(sys.executable), + ), 'language': 'system', 'files': r'\.py$', }, @@ -658,9 +932,9 @@ def test_local_hook_passes(cap_out, store, repo_with_passing_hook): } add_config_to_repo(repo_with_passing_hook, config) - with io.open('dummy.py', 'w') as staged_file: + with open('placeholder.py', 'w') as staged_file: staged_file.write('"""TODO: something"""\n') - cmd_output('git', 'add', 'dummy.py') + cmd_output('git', 'add', 'placeholder.py') _test_run( cap_out, @@ -685,9 +959,9 @@ def test_local_hook_fails(cap_out, store, repo_with_passing_hook): } add_config_to_repo(repo_with_passing_hook, config) - with io.open('dummy.py', 'w') as staged_file: + with open('placeholder.py', 'w') as staged_file: staged_file.write('"""TODO: something"""\n') - cmd_output('git', 'add', 'dummy.py') + cmd_output('git', 'add', 'placeholder.py') _test_run( cap_out, @@ -700,32 +974,6 @@ def test_local_hook_fails(cap_out, store, repo_with_passing_hook): ) -def test_pcre_deprecation_warning(cap_out, store, repo_with_passing_hook): - config = { - 'repo': 'local', - 'hooks': [{ - 'id': 'pcre-hook', - 'name': 'pcre-hook', - 'language': 'pcre', - 'entry': '.', - }], - } - add_config_to_repo(repo_with_passing_hook, config) - - _test_run( - cap_out, - store, - repo_with_passing_hook, - opts={}, - expected_outputs=[ - b'[WARNING] `pcre-hook` (from local) uses the deprecated ' - b'pcre language.', - ], - expected_ret=0, - stage=False, - ) - - def test_meta_hook_passes(cap_out, store, repo_with_passing_hook): add_config_to_repo(repo_with_passing_hook, sample_meta_config()) @@ -755,6 +1003,16 @@ def test_error_with_unstaged_config(cap_out, store, modified_config_repo): assert ret == 1 +def test_commit_msg_missing_filename(cap_out, store, repo_with_passing_hook): + args = run_opts(hook_stage='commit-msg') + ret, printed = _do_run(cap_out, store, repo_with_passing_hook, args) + assert ret == 1 + assert printed == ( + b'[ERROR] `--commit-msg-filename` is required for ' + b'`--hook-stage commit-msg`\n' + ) + + @pytest.mark.parametrize( 'opts', (run_opts(all_files=True), run_opts(files=[C.CONFIG_FILE])), ) @@ -779,7 +1037,7 @@ def test_files_running_subdir(repo_with_passing_hook, tempdir_factory): '--files', 'foo.py', tempdir_factory=tempdir_factory, ) - assert 'subdir/foo.py'.replace('/', os.sep) in stdout + assert 'subdir/foo.py' in stdout @pytest.mark.parametrize( @@ -818,11 +1076,90 @@ def test_fail_fast(cap_out, store, repo_with_failing_hook): assert printed.count(b'Failing hook') == 1 +def test_fail_fast_per_hook(cap_out, store, repo_with_failing_hook): + with modify_config() as config: + # More than one hook + config['repos'][0]['hooks'] *= 2 + config['repos'][0]['hooks'][0]['fail_fast'] = True + stage_a_file() + + ret, printed = _do_run(cap_out, store, repo_with_failing_hook, run_opts()) + # it should have only run one hook + assert printed.count(b'Failing hook') == 1 + + +def test_fail_fast_not_prev_failures(cap_out, store, repo_with_failing_hook): + with modify_config() as config: + config['repos'].append({ + 'repo': 'meta', + 'hooks': [ + {'id': 'identity', 'fail_fast': True}, + {'id': 'identity', 'name': 'run me!'}, + ], + }) + stage_a_file() + + ret, printed = _do_run(cap_out, store, repo_with_failing_hook, run_opts()) + # should still run the last hook since the `fail_fast` one didn't fail + assert printed.count(b'run me!') == 1 + + +def test_fail_fast_run_arg(cap_out, store, repo_with_failing_hook): + with modify_config() as config: + # More than one hook to demonstrate early exit + config['repos'][0]['hooks'] *= 2 + stage_a_file() + + ret, printed = _do_run( + cap_out, store, repo_with_failing_hook, run_opts(fail_fast=True), + ) + # it should have only run one hook due to the CLI flag + assert printed.count(b'Failing hook') == 1 + + def test_classifier_removes_dne(): classifier = Classifier(('this_file_does_not_exist',)) assert classifier.filenames == [] +def test_classifier_normalizes_filenames_on_windows_to_forward_slashes(tmpdir): + with tmpdir.as_cwd(): + tmpdir.join('a/b/c').ensure() + with mock.patch.object(os, 'altsep', '/'): + with mock.patch.object(os, 'sep', '\\'): + classifier = Classifier.from_config((r'a\b\c',), '', '^$') + assert classifier.filenames == ['a/b/c'] + + +def test_classifier_does_not_normalize_backslashes_non_windows(tmpdir): + with mock.patch.object(os.path, 'lexists', return_value=True): + with mock.patch.object(os, 'altsep', None): + with mock.patch.object(os, 'sep', '/'): + classifier = Classifier.from_config((r'a/b\c',), '', '^$') + assert classifier.filenames == [r'a/b\c'] + + +def test_classifier_empty_types_or(tmpdir): + tmpdir.join('bar').ensure() + os.symlink(tmpdir.join('bar'), tmpdir.join('foo')) + with tmpdir.as_cwd(): + classifier = Classifier(('foo', 'bar')) + for_symlink = classifier.by_types( + classifier.filenames, + types=['symlink'], + types_or=[], + exclude_types=[], + ) + for_file = classifier.by_types( + classifier.filenames, + types=['file'], + types_or=[], + exclude_types=[], + ) + assert tuple(for_symlink) == ('foo',) + assert tuple(for_file) == ('bar',) + + @pytest.fixture def some_filenames(): return ( @@ -834,34 +1171,33 @@ def some_filenames(): def test_include_exclude_base_case(some_filenames): ret = filter_by_include_exclude(some_filenames, '', '^$') - assert ret == [ + assert tuple(ret) == ( '.pre-commit-hooks.yaml', 'pre_commit/git.py', 'pre_commit/main.py', - ] + ) -@xfailif_no_symlink # pragma: windows no cover def test_matches_broken_symlink(tmpdir): with tmpdir.as_cwd(): os.symlink('does-not-exist', 'link') ret = filter_by_include_exclude({'link'}, '', '^$') - assert ret == ['link'] + assert tuple(ret) == ('link',) def test_include_exclude_total_match(some_filenames): ret = filter_by_include_exclude(some_filenames, r'^.*\.py$', '^$') - assert ret == ['pre_commit/git.py', 'pre_commit/main.py'] + assert tuple(ret) == ('pre_commit/git.py', 'pre_commit/main.py') def test_include_exclude_does_search_instead_of_match(some_filenames): ret = filter_by_include_exclude(some_filenames, r'\.yaml$', '^$') - assert ret == ['.pre-commit-hooks.yaml'] + assert tuple(ret) == ('.pre-commit-hooks.yaml',) def test_include_exclude_exclude_removes_files(some_filenames): ret = filter_by_include_exclude(some_filenames, '', r'\.py$') - assert ret == ['.pre-commit-hooks.yaml'] + assert tuple(ret) == ('.pre-commit-hooks.yaml',) def test_args_hook_only(cap_out, store, repo_with_passing_hook): @@ -869,11 +1205,14 @@ def test_args_hook_only(cap_out, store, repo_with_passing_hook): 'repo': 'local', 'hooks': [ { - 'id': 'flake8', - 'name': 'flake8', - 'entry': "'{}' -m flake8".format(sys.executable), + 'id': 'identity-copy', + 'name': 'identity-copy', + 'entry': '{} -m pre_commit.meta_hooks.identity'.format( + shlex.quote(sys.executable), + ), 'language': 'system', - 'stages': ['commit'], + 'files': r'\.py$', + 'stages': ['pre-commit'], }, { 'id': 'do_not_commit', @@ -891,4 +1230,19 @@ def test_args_hook_only(cap_out, store, repo_with_passing_hook): repo_with_passing_hook, run_opts(hook='do_not_commit'), ) - assert b'flake8' not in printed + assert b'identity-copy' not in printed + + +def test_skipped_without_any_setup_for_post_checkout(in_git_dir, store): + environ = {'_PRE_COMMIT_SKIP_POST_CHECKOUT': '1'} + opts = run_opts(hook_stage='post-checkout') + assert run(C.CONFIG_FILE, store, opts, environ=environ) == 0 + + +def test_pre_commit_env_variable_set(cap_out, store, repo_with_passing_hook): + args = run_opts() + environ: MutableMapping[str, str] = {} + ret, printed = _do_run( + cap_out, store, repo_with_passing_hook, args, environ, + ) + assert environ['PRE_COMMIT'] == '1' diff --git a/tests/commands/sample_config_test.py b/tests/commands/sample_config_test.py index 83942a4f..cf56e98c 100644 --- a/tests/commands/sample_config_test.py +++ b/tests/commands/sample_config_test.py @@ -1,5 +1,4 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations from pre_commit.commands.sample_config import sample_config @@ -13,7 +12,7 @@ def test_sample_config(capsys): # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v2.0.0 + rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer diff --git a/tests/commands/try_repo_test.py b/tests/commands/try_repo_test.py index d9a0401a..c5f891ea 100644 --- a/tests/commands/try_repo_test.py +++ b/tests/commands/try_repo_test.py @@ -1,8 +1,11 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os.path import re +import time +from unittest import mock + +import re_assert from pre_commit import git from pre_commit.commands.try_repo import try_repo @@ -21,9 +24,8 @@ def try_repo_opts(repo, ref=None, **kwargs): def _get_out(cap_out): - out = cap_out.get().replace('\r\n', '\n') - out = re.sub(r'\[INFO\].+\n', '', out) - start, using_config, config, rest = out.split('=' * 79 + '\n') + out = re.sub(r'\[INFO\].+\n', '', cap_out.get()) + start, using_config, config, rest = out.split(f'{"=" * 79}\n') assert using_config == 'Using config:\n' return start, config, rest @@ -41,10 +43,11 @@ def _run_try_repo(tempdir_factory, **kwargs): def test_try_repo_repo_only(cap_out, tempdir_factory): - _run_try_repo(tempdir_factory, verbose=True) + with mock.patch.object(time, 'monotonic', return_value=0.0): + _run_try_repo(tempdir_factory, verbose=True) start, config, rest = _get_out(cap_out) assert start == '' - assert re.match( + config_pattern = re_assert.Matches( '^repos:\n' '- repo: .+\n' ' rev: .+\n' @@ -52,32 +55,38 @@ def test_try_repo_repo_only(cap_out, tempdir_factory): ' - id: bash_hook\n' ' - id: bash_hook2\n' ' - id: bash_hook3\n$', - config, - ) - assert rest == ( - '[bash_hook] Bash hook................................(no files to check)Skipped\n' # noqa: E501 - '[bash_hook2] Bash hook...................................................Passed\n' # noqa: E501 - 'hookid: bash_hook2\n' - '\n' - 'test-file\n' - '\n' - '[bash_hook3] Bash hook...............................(no files to check)Skipped\n' # noqa: E501 ) + config_pattern.assert_matches(config) + assert rest == '''\ +Bash hook............................................(no files to check)Skipped +- hook id: bash_hook +Bash hook................................................................Passed +- hook id: bash_hook2 +- duration: 0s + +test-file + +Bash hook............................................(no files to check)Skipped +- hook id: bash_hook3 +''' def test_try_repo_with_specific_hook(cap_out, tempdir_factory): _run_try_repo(tempdir_factory, hook='bash_hook', verbose=True) start, config, rest = _get_out(cap_out) assert start == '' - assert re.match( + config_pattern = re_assert.Matches( '^repos:\n' '- repo: .+\n' ' rev: .+\n' ' hooks:\n' ' - id: bash_hook\n$', - config, ) - assert rest == '[bash_hook] Bash hook................................(no files to check)Skipped\n' # noqa: E501 + config_pattern.assert_matches(config) + assert rest == '''\ +Bash hook............................................(no files to check)Skipped +- hook id: bash_hook +''' def test_try_repo_relative_path(cap_out, tempdir_factory): @@ -89,6 +98,15 @@ def test_try_repo_relative_path(cap_out, tempdir_factory): assert not try_repo(try_repo_opts(relative_repo, hook='bash_hook')) +def test_try_repo_bare_repo(cap_out, tempdir_factory): + repo = make_repo(tempdir_factory, 'modified_file_returns_zero_repo') + with cwd(git_dir(tempdir_factory)): + _add_test_file() + bare_repo = os.path.join(repo, '.git') + # previously crashed attempting modification changes + assert not try_repo(try_repo_opts(bare_repo, hook='bash_hook')) + + def test_try_repo_specific_revision(cap_out, tempdir_factory): repo = make_repo(tempdir_factory, 'script_hooks_repo') ref = git.head_rev(repo) @@ -114,14 +132,14 @@ def test_try_repo_uncommitted_changes(cap_out, tempdir_factory): start, config, rest = _get_out(cap_out) assert start == '[WARNING] Creating temporary repo with uncommitted changes...\n' # noqa: E501 - assert re.match( + config_pattern = re_assert.Matches( '^repos:\n' '- repo: .+shadow-repo\n' ' rev: .+\n' ' hooks:\n' ' - id: bash_hook\n$', - config, ) + config_pattern.assert_matches(config) assert rest == 'modified name!...........................................................Passed\n' # noqa: E501 diff --git a/tests/commands/validate_config_test.py b/tests/commands/validate_config_test.py new file mode 100644 index 00000000..a475cd81 --- /dev/null +++ b/tests/commands/validate_config_test.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +import logging + +from pre_commit.commands.validate_config import validate_config + + +def test_validate_config_ok(): + assert not validate_config(('.pre-commit-config.yaml',)) + + +def test_validate_warn_on_unknown_keys_at_repo_level(tmpdir, caplog): + f = tmpdir.join('cfg.yaml') + f.write( + 'repos:\n' + '- repo: https://gitlab.com/pycqa/flake8\n' + ' rev: 3.7.7\n' + ' hooks:\n' + ' - id: flake8\n' + ' args: [--some-args]\n', + ) + ret_val = validate_config((f.strpath,)) + assert not ret_val + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + 'Unexpected key(s) present on https://gitlab.com/pycqa/flake8: ' + 'args', + ), + ] + + +def test_validate_warn_on_unknown_keys_at_top_level(tmpdir, caplog): + f = tmpdir.join('cfg.yaml') + f.write( + 'repos:\n' + '- repo: https://gitlab.com/pycqa/flake8\n' + ' rev: 3.7.7\n' + ' hooks:\n' + ' - id: flake8\n' + 'foo:\n' + ' id: 1.0.0\n', + ) + ret_val = validate_config((f.strpath,)) + assert not ret_val + assert caplog.record_tuples == [ + ( + 'pre_commit', + logging.WARNING, + 'Unexpected key(s) present at root: foo', + ), + ] + + +def test_mains_not_ok(tmpdir): + not_yaml = tmpdir.join('f.notyaml') + not_yaml.write('{') + not_schema = tmpdir.join('notconfig.yaml') + not_schema.write('{}') + + assert validate_config(('does-not-exist',)) + assert validate_config((not_yaml.strpath,)) + assert validate_config((not_schema.strpath,)) diff --git a/tests/commands/validate_manifest_test.py b/tests/commands/validate_manifest_test.py new file mode 100644 index 00000000..a4bc8ac0 --- /dev/null +++ b/tests/commands/validate_manifest_test.py @@ -0,0 +1,18 @@ +from __future__ import annotations + +from pre_commit.commands.validate_manifest import validate_manifest + + +def test_validate_manifest_ok(): + assert not validate_manifest(('.pre-commit-hooks.yaml',)) + + +def test_not_ok(tmpdir): + not_yaml = tmpdir.join('f.notyaml') + not_yaml.write('{') + not_schema = tmpdir.join('notconfig.yaml') + not_schema.write('{}') + + assert validate_manifest(('does-not-exist',)) + assert validate_manifest((not_yaml.strpath,)) + assert validate_manifest((not_schema.strpath,)) diff --git a/tests/conftest.py b/tests/conftest.py index 635ea39a..8c9cd14d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,14 +1,11 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import functools import io -import logging import os.path +from unittest import mock -import mock import pytest -import six from pre_commit import output from pre_commit.envcontext import envcontext @@ -23,32 +20,14 @@ from testing.util import cwd from testing.util import git_commit -@pytest.fixture(autouse=True) -def no_warnings(recwarn): - yield - warnings = [] - for warning in recwarn: # pragma: no cover - message = str(warning.message) - # ImportWarning: Not importing directory '...' missing __init__(.py) - if not ( - isinstance(warning.message, ImportWarning) and - message.startswith('Not importing directory ') and - ' missing __init__' in message - ): - warnings.append( - '{}:{} {}'.format(warning.filename, warning.lineno, message), - ) - assert not warnings - - @pytest.fixture def tempdir_factory(tmpdir): - class TmpdirFactory(object): + class TmpdirFactory: def __init__(self): self.tmpdir_count = 0 def get(self): - path = tmpdir.join(six.text_type(self.tmpdir_count)).strpath + path = tmpdir.join(str(self.tmpdir_count)).strpath self.tmpdir_count += 1 os.mkdir(path) return path @@ -73,29 +52,29 @@ def in_git_dir(tmpdir): def _make_conflict(): cmd_output('git', 'checkout', 'origin/master', '-b', 'foo') - with io.open('conflict_file', 'w') as conflict_file: + with open('conflict_file', 'w') as conflict_file: conflict_file.write('herp\nderp\n') cmd_output('git', 'add', 'conflict_file') - with io.open('foo_only_file', 'w') as foo_only_file: + with open('foo_only_file', 'w') as foo_only_file: foo_only_file.write('foo') cmd_output('git', 'add', 'foo_only_file') git_commit(msg=_make_conflict.__name__) cmd_output('git', 'checkout', 'origin/master', '-b', 'bar') - with io.open('conflict_file', 'w') as conflict_file: + with open('conflict_file', 'w') as conflict_file: conflict_file.write('harp\nddrp\n') cmd_output('git', 'add', 'conflict_file') - with io.open('bar_only_file', 'w') as bar_only_file: + with open('bar_only_file', 'w') as bar_only_file: bar_only_file.write('bar') cmd_output('git', 'add', 'bar_only_file') git_commit(msg=_make_conflict.__name__) - cmd_output('git', 'merge', 'foo', retcode=None) + cmd_output('git', 'merge', 'foo', check=False) @pytest.fixture def in_merge_conflict(tempdir_factory): path = make_consuming_repo(tempdir_factory, 'script_hooks_repo') - open(os.path.join(path, 'dummy'), 'a').close() - cmd_output('git', 'add', 'dummy', cwd=path) + open(os.path.join(path, 'placeholder'), 'a').close() + cmd_output('git', 'add', 'placeholder', cwd=path) git_commit(msg=in_merge_conflict.__name__, cwd=path) conflict_path = tempdir_factory.get() @@ -145,14 +124,14 @@ def prepare_commit_msg_repo(tempdir_factory): 'hooks': [{ 'id': 'add-signoff', 'name': 'Add "Signed off by:"', - 'entry': './{}'.format(script_name), + 'entry': f'./{script_name}', 'language': 'script', 'stages': ['prepare-commit-msg'], }], } write_config(path, config) with cwd(path): - with io.open(script_name, 'w') as script_file: + with open(script_name, 'w') as script_file: script_file.write( '#!/usr/bin/env bash\n' 'set -eu\n' @@ -223,60 +202,33 @@ def store(tempdir_factory): yield Store(os.path.join(tempdir_factory.get(), '.pre-commit')) -@pytest.fixture -def log_info_mock(): - with mock.patch.object(logging.getLogger('pre_commit'), 'info') as mck: - yield mck - - -class FakeStream(object): - def __init__(self): - self.data = io.BytesIO() - - def write(self, s): - self.data.write(s) - - def flush(self): - pass - - -class Fixture(object): - def __init__(self, stream): +class Fixture: + def __init__(self, stream: io.BytesIO) -> None: self._stream = stream - def get_bytes(self): + def get_bytes(self) -> bytes: """Get the output as-if no encoding occurred""" - data = self._stream.data.getvalue() - self._stream.data.seek(0) - self._stream.data.truncate() - return data + data = self._stream.getvalue() + self._stream.seek(0) + self._stream.truncate() + return data.replace(b'\r\n', b'\n') - def get(self): + def get(self) -> str: """Get the output assuming it was written as UTF-8 bytes""" - return self.get_bytes().decode('UTF-8') + return self.get_bytes().decode() @pytest.fixture def cap_out(): - stream = FakeStream() + stream = io.BytesIO() write = functools.partial(output.write, stream=stream) - write_line = functools.partial(output.write_line, stream=stream) - with mock.patch.object(output, 'write', write): - with mock.patch.object(output, 'write_line', write_line): - yield Fixture(stream) - - -@pytest.fixture -def fake_log_handler(): - handler = mock.Mock(level=logging.INFO) - logger = logging.getLogger('pre_commit') - logger.addHandler(handler) - yield handler - logger.removeHandler(handler) + write_line_b = functools.partial(output.write_line_b, stream=stream) + with mock.patch.multiple(output, write=write, write_line_b=write_line_b): + yield Fixture(stream) @pytest.fixture(scope='session', autouse=True) def set_git_templatedir(tmpdir_factory): tdir = str(tmpdir_factory.mktemp('git_template_dir')) - with envcontext([('GIT_TEMPLATE_DIR', tdir)]): + with envcontext((('GIT_TEMPLATE_DIR', tdir),)): yield diff --git a/tests/envcontext_test.py b/tests/envcontext_test.py index c03e9431..c82d3267 100644 --- a/tests/envcontext_test.py +++ b/tests/envcontext_test.py @@ -1,9 +1,8 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os +from unittest import mock -import mock import pytest from pre_commit.envcontext import envcontext @@ -11,12 +10,7 @@ from pre_commit.envcontext import UNSET from pre_commit.envcontext import Var -def _test(**kwargs): - before = kwargs.pop('before') - patch = kwargs.pop('patch') - expected = kwargs.pop('expected') - assert not kwargs - +def _test(*, before, patch, expected): env = before.copy() with envcontext(patch, _env=env): assert env == expected @@ -94,16 +88,16 @@ def test_exception_safety(): class MyError(RuntimeError): pass - env = {} + env = {'hello': 'world'} with pytest.raises(MyError): - with envcontext([('foo', 'bar')], _env=env): + with envcontext((('foo', 'bar'),), _env=env): raise MyError() - assert env == {} + assert env == {'hello': 'world'} def test_integration_os_environ(): with mock.patch.dict(os.environ, {'FOO': 'bar'}, clear=True): assert os.environ == {'FOO': 'bar'} - with envcontext([('HERP', 'derp')]): + with envcontext((('HERP', 'derp'),)): assert os.environ == {'FOO': 'bar', 'HERP': 'derp'} assert os.environ == {'FOO': 'bar'} diff --git a/tests/error_handler_test.py b/tests/error_handler_test.py index 1b222f90..a79d9c1a 100644 --- a/tests/error_handler_test.py +++ b/tests/error_handler_test.py @@ -1,17 +1,19 @@ -# -*- coding: utf-8 -*- -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import io import os.path -import re +import stat import sys +from unittest import mock -import mock import pytest +import re_assert from pre_commit import error_handler +from pre_commit.errors import FatalError +from pre_commit.store import Store +from pre_commit.util import CalledProcessError from testing.util import cmd_output_mocked_pre_commit_home +from testing.util import xfailif_windows @pytest.fixture @@ -27,27 +29,30 @@ def test_error_handler_no_exception(mocked_log_and_exit): def test_error_handler_fatal_error(mocked_log_and_exit): - exc = error_handler.FatalError('just a test') + exc = FatalError('just a test') with error_handler.error_handler(): raise exc mocked_log_and_exit.assert_called_once_with( 'An error has occurred', + 1, exc, # Tested below mock.ANY, ) - assert re.match( + pattern = re_assert.Matches( r'Traceback \(most recent call last\):\n' r' File ".+pre_commit.error_handler.py", line \d+, in error_handler\n' r' yield\n' + r'( \^\^\^\^\^\n)?' r' File ".+tests.error_handler_test.py", line \d+, ' r'in test_error_handler_fatal_error\n' r' raise exc\n' - r'(pre_commit\.error_handler\.)?FatalError: just a test\n', - mocked_log_and_exit.call_args[0][2], + r'( \^\^\^\^\^\^\^\^\^\n)?' + r'(pre_commit\.errors\.)?FatalError: just a test\n', ) + pattern.assert_matches(mocked_log_and_exit.call_args[0][3]) def test_error_handler_uncaught_error(mocked_log_and_exit): @@ -57,20 +62,23 @@ def test_error_handler_uncaught_error(mocked_log_and_exit): mocked_log_and_exit.assert_called_once_with( 'An unexpected error has occurred', + 3, exc, # Tested below mock.ANY, ) - assert re.match( + pattern = re_assert.Matches( r'Traceback \(most recent call last\):\n' r' File ".+pre_commit.error_handler.py", line \d+, in error_handler\n' r' yield\n' + r'( \^\^\^\^\^\n)?' r' File ".+tests.error_handler_test.py", line \d+, ' r'in test_error_handler_uncaught_error\n' r' raise exc\n' + r'( \^\^\^\^\^\^\^\^\^\n)?' r'ValueError: another test\n', - mocked_log_and_exit.call_args[0][2], ) + pattern.assert_matches(mocked_log_and_exit.call_args[0][3]) def test_error_handler_keyboardinterrupt(mocked_log_and_exit): @@ -80,41 +88,69 @@ def test_error_handler_keyboardinterrupt(mocked_log_and_exit): mocked_log_and_exit.assert_called_once_with( 'Interrupted (^C)', + 130, exc, # Tested below mock.ANY, ) - assert re.match( + pattern = re_assert.Matches( r'Traceback \(most recent call last\):\n' r' File ".+pre_commit.error_handler.py", line \d+, in error_handler\n' r' yield\n' + r'( \^\^\^\^\^\n)?' r' File ".+tests.error_handler_test.py", line \d+, ' r'in test_error_handler_keyboardinterrupt\n' r' raise exc\n' + r'( \^\^\^\^\^\^\^\^\^\n)?' r'KeyboardInterrupt\n', - mocked_log_and_exit.call_args[0][2], ) + pattern.assert_matches(mocked_log_and_exit.call_args[0][3]) def test_log_and_exit(cap_out, mock_store_dir): - with pytest.raises(SystemExit): - error_handler._log_and_exit( - 'msg', error_handler.FatalError('hai'), "I'm a stacktrace", - ) + tb = ( + 'Traceback (most recent call last):\n' + ' File "", line 2, in \n' + 'pre_commit.errors.FatalError: hai\n' + ) + + with pytest.raises(SystemExit) as excinfo: + error_handler._log_and_exit('msg', 1, FatalError('hai'), tb) + assert excinfo.value.code == 1 printed = cap_out.get() log_file = os.path.join(mock_store_dir, 'pre-commit.log') - assert printed == ( - 'msg: FatalError: hai\n' - 'Check the log at {}\n'.format(log_file) - ) + assert printed == f'msg: FatalError: hai\nCheck the log at {log_file}\n' assert os.path.exists(log_file) - with io.open(log_file) as f: - assert f.read() == ( - 'msg: FatalError: hai\n' - "I'm a stacktrace\n" + with open(log_file) as f: + logged = f.read() + pattern = re_assert.Matches( + r'^### version information\n' + r'\n' + r'```\n' + r'pre-commit version: \d+\.\d+\.\d+\n' + r'git --version: git version .+\n' + r'sys.version:\n' + r'( .*\n)*' + r'sys.executable: .*\n' + r'os.name: .*\n' + r'sys.platform: .*\n' + r'```\n' + r'\n' + r'### error information\n' + r'\n' + r'```\n' + r'msg: FatalError: hai\n' + r'```\n' + r'\n' + r'```\n' + r'Traceback \(most recent call last\):\n' + r' File "", line 2, in \n' + r'pre_commit\.errors\.FatalError: hai\n' + r'```\n', ) + pattern.assert_matches(logged) def test_error_handler_non_ascii_exception(mock_store_dir): @@ -123,20 +159,62 @@ def test_error_handler_non_ascii_exception(mock_store_dir): raise ValueError('β˜ƒ') +def test_error_handler_non_utf8_exception(mock_store_dir): + with pytest.raises(SystemExit): + with error_handler.error_handler(): + raise CalledProcessError(1, ('exe',), b'error: \xa0\xe1', b'') + + +def test_error_handler_non_stringable_exception(mock_store_dir): + class C(Exception): + def __str__(self): + raise RuntimeError('not today!') + + with pytest.raises(SystemExit): + with error_handler.error_handler(): + raise C() + + def test_error_handler_no_tty(tempdir_factory): pre_commit_home = tempdir_factory.get() - output = cmd_output_mocked_pre_commit_home( - sys.executable, '-c', - 'from __future__ import unicode_literals\n' + ret, out, _ = cmd_output_mocked_pre_commit_home( + sys.executable, + '-c', 'from pre_commit.error_handler import error_handler\n' 'with error_handler():\n' ' raise ValueError("\\u2603")\n', - retcode=1, + check=False, tempdir_factory=tempdir_factory, pre_commit_home=pre_commit_home, ) + assert ret == 3 log_file = os.path.join(pre_commit_home, 'pre-commit.log') - assert output[1].replace('\r', '') == ( - 'An unexpected error has occurred: ValueError: β˜ƒ\n' - 'Check the log at {}\n'.format(log_file) + out_lines = out.splitlines() + assert out_lines[-2] == 'An unexpected error has occurred: ValueError: β˜ƒ' + assert out_lines[-1] == f'Check the log at {log_file}' + + +@xfailif_windows # pragma: win32 no cover +def test_error_handler_read_only_filesystem(mock_store_dir, cap_out, capsys): + # a better scenario would be if even the Store crash would be handled + # but realistically we're only targetting systems where the Store has + # already been set up + Store() + + write = (stat.S_IWGRP | stat.S_IWOTH | stat.S_IWUSR) + os.chmod(mock_store_dir, os.stat(mock_store_dir).st_mode & ~write) + + with pytest.raises(SystemExit): + with error_handler.error_handler(): + raise ValueError('ohai') + + output = cap_out.get() + assert output.startswith( + 'An unexpected error has occurred: ValueError: ohai\n' + 'Failed to write to log at ', ) + + # our cap_out mock is imperfect so the rest of the output goes to capsys + out, _ = capsys.readouterr() + # the things that normally go to the log file will end up here + assert '### version information' in out diff --git a/tests/git_test.py b/tests/git_test.py index 299729db..02b6ce3a 100644 --- a/tests/git_test.py +++ b/tests/git_test.py @@ -1,12 +1,11 @@ -# -*- coding: utf-8 -*- -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os.path import pytest from pre_commit import git +from pre_commit.error_handler import FatalError from pre_commit.util import cmd_output from testing.util import git_commit @@ -22,6 +21,66 @@ def test_get_root_deeper(in_git_dir): assert os.path.normcase(git.get_root()) == expected +def test_get_root_in_git_sub_dir(in_git_dir): + expected = os.path.normcase(in_git_dir.strpath) + with pytest.raises(FatalError): + with in_git_dir.join('.git/objects').ensure_dir().as_cwd(): + assert os.path.normcase(git.get_root()) == expected + + +def test_get_root_not_in_working_dir(in_git_dir): + expected = os.path.normcase(in_git_dir.strpath) + with pytest.raises(FatalError): + with in_git_dir.join('..').ensure_dir().as_cwd(): + assert os.path.normcase(git.get_root()) == expected + + +def test_in_exactly_dot_git(in_git_dir): + with in_git_dir.join('.git').as_cwd(), pytest.raises(FatalError): + git.get_root() + + +def test_get_root_bare_worktree(tmpdir): + src = tmpdir.join('src').ensure_dir() + cmd_output('git', 'init', str(src)) + git_commit(cwd=str(src)) + + bare = tmpdir.join('bare.git').ensure_dir() + cmd_output('git', 'clone', '--bare', str(src), str(bare)) + + cmd_output('git', 'worktree', 'add', 'foo', 'HEAD', cwd=bare) + + with bare.join('foo').as_cwd(): + assert git.get_root() == os.path.abspath('.') + + +def test_get_git_dir(tmpdir): + """Regression test for #1972""" + src = tmpdir.join('src').ensure_dir() + cmd_output('git', 'init', str(src)) + git_commit(cwd=str(src)) + + worktree = tmpdir.join('worktree').ensure_dir() + cmd_output('git', 'worktree', 'add', '../worktree', cwd=src) + + with worktree.as_cwd(): + assert git.get_git_dir() == src.ensure_dir( + '.git/worktrees/worktree', + ) + assert git.get_git_common_dir() == src.ensure_dir('.git') + + +def test_get_root_worktree_in_git(tmpdir): + src = tmpdir.join('src').ensure_dir() + cmd_output('git', 'init', str(src)) + git_commit(cwd=str(src)) + + cmd_output('git', 'worktree', 'add', '.git/trees/foo', 'HEAD', cwd=src) + + with src.join('.git/trees/foo').as_cwd(): + assert git.get_root() == os.path.abspath('.') + + def test_get_staged_files_deleted(in_git_dir): in_git_dir.join('test').ensure() cmd_output('git', 'add', 'test') @@ -45,7 +104,7 @@ def test_is_in_merge_conflict_submodule(in_conflicting_submodule): def test_cherry_pick_conflict(in_merge_conflict): cmd_output('git', 'merge', '--abort') foo_ref = cmd_output('git', 'rev-parse', 'foo')[1].strip() - cmd_output('git', 'cherry-pick', foo_ref, retcode=None) + cmd_output('git', 'cherry-pick', foo_ref, check=False) assert git.is_in_merge_conflict() is False @@ -82,6 +141,15 @@ def test_get_conflicted_files_unstaged_files(in_merge_conflict): assert ret == {'conflict_file'} +def test_get_conflicted_files_with_file_named_head(in_merge_conflict): + resolve_conflict() + open('HEAD', 'w').close() + cmd_output('git', 'add', 'HEAD') + + ret = set(git.get_conflicted_files()) + assert ret == {'conflict_file', 'HEAD'} + + MERGE_MSG = b"Merge branch 'foo' into bar\n\nConflicts:\n\tconflict_file\n" OTHER_MERGE_MSG = MERGE_MSG + b'\tother_conflict_file\n' @@ -104,14 +172,32 @@ def test_get_changed_files(in_git_dir): in_git_dir.join('b.txt').ensure() cmd_output('git', 'add', '.') git_commit() - files = git.get_changed_files('HEAD', 'HEAD^') + files = git.get_changed_files('HEAD^', 'HEAD') assert files == ['a.txt', 'b.txt'] # files changed in source but not in origin should not be returned - files = git.get_changed_files('HEAD^', 'HEAD') + files = git.get_changed_files('HEAD', 'HEAD^') assert files == [] +def test_get_changed_files_disparate_histories(in_git_dir): + """in modern versions of git, `...` does not fall back to full diff""" + git_commit() + in_git_dir.join('a.txt').ensure() + cmd_output('git', 'add', '.') + git_commit() + cmd_output('git', 'branch', '-m', 'branch1') + + cmd_output('git', 'checkout', '--orphan', 'branch2') + cmd_output('git', 'rm', '-rf', '.') + in_git_dir.join('a.txt').ensure() + in_git_dir.join('b.txt').ensure() + cmd_output('git', 'add', '.') + git_commit() + + assert git.get_changed_files('branch1', 'branch2') == ['b.txt'] + + @pytest.mark.parametrize( ('s', 'expected'), ( @@ -146,7 +232,7 @@ def test_staged_files_non_ascii(non_ascii_repo): def test_changed_files_non_ascii(non_ascii_repo): - ret = git.get_changed_files('HEAD', 'HEAD^') + ret = git.get_changed_files('HEAD^', 'HEAD') assert ret == ['ΠΈΠ½Ρ‚Π΅Ρ€Π²ΡŒΡŽ'] @@ -182,6 +268,11 @@ def test_no_git_env(): 'GIT_SSH': '/usr/bin/ssh', 'GIT_SSH_COMMAND': 'ssh -o', 'GIT_DIR': '/none/shall/pass', + 'GIT_CONFIG_KEY_0': 'user.name', + 'GIT_CONFIG_VALUE_0': 'anthony', + 'GIT_CONFIG_KEY_1': 'user.email', + 'GIT_CONFIG_VALUE_1': 'asottile@example.com', + 'GIT_CONFIG_COUNT': '2', } no_git_env = git.no_git_env(env) assert no_git_env == { @@ -189,4 +280,14 @@ def test_no_git_env(): 'GIT_EXEC_PATH': '/some/git/exec/path', 'GIT_SSH': '/usr/bin/ssh', 'GIT_SSH_COMMAND': 'ssh -o', + 'GIT_CONFIG_KEY_0': 'user.name', + 'GIT_CONFIG_VALUE_0': 'anthony', + 'GIT_CONFIG_KEY_1': 'user.email', + 'GIT_CONFIG_VALUE_1': 'asottile@example.com', + 'GIT_CONFIG_COUNT': '2', } + + +def test_init_repo_no_hooks(tmpdir): + git.init_repo(str(tmpdir), remote='dne') + assert not tmpdir.join('.git/hooks').exists() diff --git a/tests/lang_base_test.py b/tests/lang_base_test.py new file mode 100644 index 00000000..9fac83da --- /dev/null +++ b/tests/lang_base_test.py @@ -0,0 +1,178 @@ +from __future__ import annotations + +import os.path +import sys +from unittest import mock + +import pytest + +import pre_commit.constants as C +from pre_commit import lang_base +from pre_commit import parse_shebang +from pre_commit import xargs +from pre_commit.prefix import Prefix +from pre_commit.util import CalledProcessError + + +@pytest.fixture +def find_exe_mck(): + with mock.patch.object(parse_shebang, 'find_executable') as mck: + yield mck + + +@pytest.fixture +def homedir_mck(): + def fake_expanduser(pth): + assert pth == '~' + return os.path.normpath('/home/me') + + with mock.patch.object(os.path, 'expanduser', fake_expanduser): + yield + + +def test_exe_exists_does_not_exist(find_exe_mck, homedir_mck): + find_exe_mck.return_value = None + assert lang_base.exe_exists('ruby') is False + + +def test_exe_exists_exists(find_exe_mck, homedir_mck): + find_exe_mck.return_value = os.path.normpath('/usr/bin/ruby') + assert lang_base.exe_exists('ruby') is True + + +def test_exe_exists_false_if_shim(find_exe_mck, homedir_mck): + find_exe_mck.return_value = os.path.normpath('/foo/shims/ruby') + assert lang_base.exe_exists('ruby') is False + + +def test_exe_exists_false_if_homedir(find_exe_mck, homedir_mck): + find_exe_mck.return_value = os.path.normpath('/home/me/somedir/ruby') + assert lang_base.exe_exists('ruby') is False + + +def test_exe_exists_commonpath_raises_ValueError(find_exe_mck, homedir_mck): + find_exe_mck.return_value = os.path.normpath('/usr/bin/ruby') + with mock.patch.object(os.path, 'commonpath', side_effect=ValueError): + assert lang_base.exe_exists('ruby') is True + + +def test_exe_exists_true_when_homedir_is_slash(find_exe_mck): + find_exe_mck.return_value = os.path.normpath('/usr/bin/ruby') + with mock.patch.object(os.path, 'expanduser', return_value=os.sep): + assert lang_base.exe_exists('ruby') is True + + +def test_basic_get_default_version(): + assert lang_base.basic_get_default_version() == C.DEFAULT + + +def test_basic_health_check(): + assert lang_base.basic_health_check(Prefix('.'), 'default') is None + + +def test_failed_setup_command_does_not_unicode_error(): + script = ( + 'import sys\n' + "sys.stderr.buffer.write(b'\\x81\\xfe')\n" + 'raise SystemExit(1)\n' + ) + + # an assertion that this does not raise `UnicodeError` + with pytest.raises(CalledProcessError): + lang_base.setup_cmd(Prefix('.'), (sys.executable, '-c', script)) + + +def test_environment_dir(tmp_path): + ret = lang_base.environment_dir(Prefix(tmp_path), 'langenv', 'default') + assert ret == f'{tmp_path}{os.sep}langenv-default' + + +def test_assert_version_default(): + with pytest.raises(AssertionError) as excinfo: + lang_base.assert_version_default('lang', '1.2.3') + msg, = excinfo.value.args + assert msg == ( + 'for now, pre-commit requires system-installed lang -- ' + 'you selected `language_version: 1.2.3`' + ) + + +def test_assert_no_additional_deps(): + with pytest.raises(AssertionError) as excinfo: + lang_base.assert_no_additional_deps('lang', ['hmmm']) + msg, = excinfo.value.args + assert msg == ( + 'for now, pre-commit does not support additional_dependencies for ' + 'lang -- ' + "you selected `additional_dependencies: ['hmmm']`" + ) + + +def test_no_env_noop(tmp_path): + before = os.environ.copy() + with lang_base.no_env(Prefix(tmp_path), '1.2.3'): + inside = os.environ.copy() + after = os.environ.copy() + assert before == inside == after + + +@pytest.fixture +def cpu_count_mck(): + with mock.patch.object(xargs, 'cpu_count', return_value=4): + yield + + +@pytest.mark.parametrize( + ('var', 'expected'), + ( + ('PRE_COMMIT_NO_CONCURRENCY', 1), + ('TRAVIS', 2), + (None, 4), + ), +) +def test_target_concurrency(cpu_count_mck, var, expected): + with mock.patch.dict(os.environ, {var: '1'} if var else {}, clear=True): + assert lang_base.target_concurrency() == expected + + +def test_shuffled_is_deterministic(): + seq = [str(i) for i in range(10)] + expected = ['4', '0', '5', '1', '8', '6', '2', '3', '7', '9'] + assert lang_base._shuffled(seq) == expected + + +def test_xargs_require_serial_is_not_shuffled(): + ret, out = lang_base.run_xargs( + ('echo',), [str(i) for i in range(10)], + require_serial=True, + color=False, + ) + assert ret == 0 + assert out.strip() == b'0 1 2 3 4 5 6 7 8 9' + + +def test_basic_run_hook(tmp_path): + ret, out = lang_base.basic_run_hook( + Prefix(tmp_path), + 'echo hi', + ['hello'], + ['file', 'file', 'file'], + is_local=False, + require_serial=False, + color=False, + ) + assert ret == 0 + out = out.replace(b'\r\n', b'\n') + assert out == b'hi hello file file file\n' + + +def test_hook_cmd(): + assert lang_base.hook_cmd('echo hi', ()) == ('echo', 'hi') + + +def test_hook_cmd_hazmat(): + ret = lang_base.hook_cmd('pre-commit hazmat cd a echo -- b', ()) + assert ret == ( + sys.executable, '-m', 'pre_commit.commands.hazmat', + 'cd', 'a', 'echo', '--', 'b', + ) diff --git a/tests/languages/all_test.py b/tests/languages/all_test.py deleted file mode 100644 index 96754419..00000000 --- a/tests/languages/all_test.py +++ /dev/null @@ -1,58 +0,0 @@ -from __future__ import unicode_literals - -import functools -import inspect - -import pytest -import six - -from pre_commit.languages.all import all_languages -from pre_commit.languages.all import languages - - -if six.PY2: # pragma: no cover - ArgSpec = functools.partial( - inspect.ArgSpec, varargs=None, keywords=None, defaults=None, - ) - getargspec = inspect.getargspec -else: # pragma: no cover - ArgSpec = functools.partial( - inspect.FullArgSpec, varargs=None, varkw=None, defaults=None, - kwonlyargs=[], kwonlydefaults=None, annotations={}, - ) - getargspec = inspect.getfullargspec - - -@pytest.mark.parametrize('language', all_languages) -def test_install_environment_argspec(language): - expected_argspec = ArgSpec( - args=['prefix', 'version', 'additional_dependencies'], - ) - argspec = getargspec(languages[language].install_environment) - assert argspec == expected_argspec - - -@pytest.mark.parametrize('language', all_languages) -def test_ENVIRONMENT_DIR(language): - assert hasattr(languages[language], 'ENVIRONMENT_DIR') - - -@pytest.mark.parametrize('language', all_languages) -def test_run_hook_argpsec(language): - expected_argspec = ArgSpec(args=['hook', 'file_args']) - argspec = getargspec(languages[language].run_hook) - assert argspec == expected_argspec - - -@pytest.mark.parametrize('language', all_languages) -def test_get_default_version_argspec(language): - expected_argspec = ArgSpec(args=[]) - argspec = getargspec(languages[language].get_default_version) - assert argspec == expected_argspec - - -@pytest.mark.parametrize('language', all_languages) -def test_healthy_argspec(language): - expected_argspec = ArgSpec(args=['prefix', 'language_version']) - argspec = getargspec(languages[language].healthy) - assert argspec == expected_argspec diff --git a/tests/languages/conda_test.py b/tests/languages/conda_test.py new file mode 100644 index 00000000..83aaebed --- /dev/null +++ b/tests/languages/conda_test.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +import os.path + +import pytest + +from pre_commit import envcontext +from pre_commit.languages import conda +from pre_commit.store import _make_local_repo +from testing.language_helpers import run_language + + +@pytest.mark.parametrize( + ('ctx', 'expected'), + ( + pytest.param( + ( + ('PRE_COMMIT_USE_MICROMAMBA', envcontext.UNSET), + ('PRE_COMMIT_USE_MAMBA', envcontext.UNSET), + ), + 'conda', + id='default', + ), + pytest.param( + ( + ('PRE_COMMIT_USE_MICROMAMBA', '1'), + ('PRE_COMMIT_USE_MAMBA', ''), + ), + 'micromamba', + id='default', + ), + pytest.param( + ( + ('PRE_COMMIT_USE_MICROMAMBA', ''), + ('PRE_COMMIT_USE_MAMBA', '1'), + ), + 'mamba', + id='default', + ), + ), +) +def test_conda_exe(ctx, expected): + with envcontext.envcontext(ctx): + assert conda._conda_exe() == expected + + +def test_conda_language(tmp_path): + environment_yml = '''\ +channels: [conda-forge, defaults] +dependencies: [python, pip] +''' + tmp_path.joinpath('environment.yml').write_text(environment_yml) + + ret, out = run_language( + tmp_path, + conda, + 'python -c "import sys; print(sys.prefix)"', + ) + assert ret == 0 + assert os.path.basename(out.strip()) == b'conda-default' + + +def test_conda_additional_deps(tmp_path): + _make_local_repo(tmp_path) + + ret = run_language( + tmp_path, + conda, + 'python -c "import botocore; print(1)"', + deps=('botocore',), + ) + assert ret == (0, b'1\n') diff --git a/tests/languages/coursier_test.py b/tests/languages/coursier_test.py new file mode 100644 index 00000000..dbb746ca --- /dev/null +++ b/tests/languages/coursier_test.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +import pytest + +from pre_commit.errors import FatalError +from pre_commit.languages import coursier +from testing.language_helpers import run_language + + +def test_coursier_hook(tmp_path): + echo_java_json = '''\ +{ + "repositories": ["central"], + "dependencies": ["io.get-coursier:echo:latest.stable"] +} +''' + + channel_dir = tmp_path.joinpath('.pre-commit-channel') + channel_dir.mkdir() + channel_dir.joinpath('echo-java.json').write_text(echo_java_json) + + ret = run_language( + tmp_path, + coursier, + 'echo-java', + args=('Hello', 'World', 'from', 'coursier'), + ) + assert ret == (0, b'Hello World from coursier\n') + + +def test_coursier_hook_additional_dependencies(tmp_path): + ret = run_language( + tmp_path, + coursier, + 'scalafmt --version', + deps=('scalafmt:3.6.1',), + ) + assert ret == (0, b'scalafmt 3.6.1\n') + + +def test_error_if_no_deps_or_channel(tmp_path): + with pytest.raises(FatalError) as excinfo: + run_language(tmp_path, coursier, 'dne') + msg, = excinfo.value.args + assert msg == 'expected .pre-commit-channel dir or additional_dependencies' diff --git a/tests/languages/dart_test.py b/tests/languages/dart_test.py new file mode 100644 index 00000000..213d888e --- /dev/null +++ b/tests/languages/dart_test.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +import re_assert + +from pre_commit.languages import dart +from pre_commit.store import _make_local_repo +from testing.language_helpers import run_language + + +def test_dart(tmp_path): + pubspec_yaml = '''\ +environment: + sdk: '>=2.12.0 <4.0.0' + +name: hello_world_dart + +executables: + hello-world-dart: + +dependencies: + ansicolor: ^2.0.1 +''' + hello_world_dart_dart = '''\ +import 'package:ansicolor/ansicolor.dart'; + +void main() { + AnsiPen pen = new AnsiPen()..red(); + print("hello hello " + pen("world")); +} +''' + tmp_path.joinpath('pubspec.yaml').write_text(pubspec_yaml) + bin_dir = tmp_path.joinpath('bin') + bin_dir.mkdir() + bin_dir.joinpath('hello-world-dart.dart').write_text(hello_world_dart_dart) + + expected = (0, b'hello hello world\n') + assert run_language(tmp_path, dart, 'hello-world-dart') == expected + + +def test_dart_additional_deps(tmp_path): + _make_local_repo(str(tmp_path)) + + ret = run_language( + tmp_path, + dart, + 'hello-world-dart', + deps=('hello_world_dart',), + ) + assert ret == (0, b'hello hello world\n') + + +def test_dart_additional_deps_versioned(tmp_path): + _make_local_repo(str(tmp_path)) + + ret, out = run_language( + tmp_path, + dart, + 'secure-random -l 4 -b 16', + deps=('encrypt:5.0.0',), + ) + assert ret == 0 + re_assert.Matches('^[a-f0-9]{8}\n$').assert_matches(out.decode()) diff --git a/tests/languages/docker_image_test.py b/tests/languages/docker_image_test.py new file mode 100644 index 00000000..4f720600 --- /dev/null +++ b/tests/languages/docker_image_test.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import pytest + +from pre_commit.languages import docker_image +from pre_commit.util import cmd_output_b +from testing.language_helpers import run_language +from testing.util import xfailif_windows + + +@pytest.fixture(autouse=True, scope='module') +def _ensure_image_available(): + cmd_output_b('docker', 'run', '--rm', 'ubuntu:22.04', 'echo') + + +@xfailif_windows # pragma: win32 no cover +def test_docker_image_hook_via_entrypoint(tmp_path): + ret = run_language( + tmp_path, + docker_image, + '--entrypoint echo ubuntu:22.04', + args=('hello hello world',), + ) + assert ret == (0, b'hello hello world\n') + + +@xfailif_windows # pragma: win32 no cover +def test_docker_image_hook_via_args(tmp_path): + ret = run_language( + tmp_path, + docker_image, + 'ubuntu:22.04 echo', + args=('hello hello world',), + ) + assert ret == (0, b'hello hello world\n') + + +@xfailif_windows # pragma: win32 no cover +def test_docker_image_color_tty(tmp_path): + ret = run_language( + tmp_path, + docker_image, + 'ubuntu:22.04', + args=('grep', '--color', 'root', '/etc/group'), + color=True, + ) + assert ret == (0, b'\x1b[01;31m\x1b[Kroot\x1b[m\x1b[K:x:0:\n') + + +@xfailif_windows # pragma: win32 no cover +def test_docker_image_no_color_no_tty(tmp_path): + ret = run_language( + tmp_path, + docker_image, + 'ubuntu:22.04', + args=('grep', '--color', 'root', '/etc/group'), + color=False, + ) + assert ret == (0, b'root:x:0:\n') diff --git a/tests/languages/docker_test.py b/tests/languages/docker_test.py index 1a96e69d..e269976f 100644 --- a/tests/languages/docker_test.py +++ b/tests/languages/docker_test.py @@ -1,26 +1,373 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import mock +import builtins +import json +import ntpath +import os.path +import posixpath +from unittest import mock + +import pytest from pre_commit.languages import docker from pre_commit.util import CalledProcessError +from testing.language_helpers import run_language +from testing.util import xfailif_windows +DOCKER_CGROUPS_V1_MOUNTINFO_EXAMPLE = b'''\ +759 717 0:52 / / rw,relatime master:300 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/PCPE5P5IVGM7CFCPJR353N3ONK:/var/lib/docker/overlay2/l/EQFSDHFAJ333VEMEJD4ZTRIZCB,upperdir=/var/lib/docker/overlay2/0d9f6bf186030d796505b87d6daa92297355e47641e283d3c09d83a7f221e462/diff,workdir=/var/lib/docker/overlay2/0d9f6bf186030d796505b87d6daa92297355e47641e283d3c09d83a7f221e462/work +760 759 0:58 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw +761 759 0:59 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +762 761 0:60 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666 +763 759 0:61 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro +764 763 0:62 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,mode=755,inode64 +765 764 0:29 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,xattr,name=systemd +766 764 0:32 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/rdma ro,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,rdma +767 764 0:33 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/cpu,cpuacct ro,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,cpu,cpuacct +768 764 0:34 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,cpuset +769 764 0:35 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/pids ro,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,pids +770 764 0:36 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,memory +771 764 0:37 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,perf_event +772 764 0:38 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/net_cls,net_prio ro,nosuid,nodev,noexec,relatime master:21 - cgroup cgroup rw,net_cls,net_prio +773 764 0:39 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime master:22 - cgroup cgroup rw,blkio +774 764 0:40 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/misc ro,nosuid,nodev,noexec,relatime master:23 - cgroup cgroup rw,misc +775 764 0:41 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/hugetlb ro,nosuid,nodev,noexec,relatime master:24 - cgroup cgroup rw,hugetlb +776 764 0:42 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime master:25 - cgroup cgroup rw,devices +777 764 0:43 /docker/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7 /sys/fs/cgroup/freezer ro,nosuid,nodev,noexec,relatime master:26 - cgroup cgroup rw,freezer +778 761 0:57 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw +779 761 0:63 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k,inode64 +780 759 8:5 /var/lib/docker/containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/sda5 rw,errors=remount-ro +781 759 8:5 /var/lib/docker/containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/hostname /etc/hostname rw,relatime - ext4 /dev/sda5 rw,errors=remount-ro +782 759 8:5 /var/lib/docker/containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/hosts /etc/hosts rw,relatime - ext4 /dev/sda5 rw,errors=remount-ro +718 761 0:60 /0 /dev/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666 +719 760 0:58 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw +720 760 0:58 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw +721 760 0:58 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw +722 760 0:58 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw +723 760 0:58 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw +724 760 0:64 / /proc/asound ro,relatime - tmpfs tmpfs ro,inode64 +725 760 0:65 / /proc/acpi ro,relatime - tmpfs tmpfs ro,inode64 +726 760 0:59 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +727 760 0:59 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +728 760 0:59 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +729 760 0:66 / /proc/scsi ro,relatime - tmpfs tmpfs ro,inode64 +730 763 0:67 / /sys/firmware ro,relatime - tmpfs tmpfs ro,inode64 +731 763 0:68 / /sys/devices/virtual/powercap ro,relatime - tmpfs tmpfs ro,inode64 +''' # noqa: E501 -def test_docker_is_running_process_error(): - with mock.patch( - 'pre_commit.languages.docker.cmd_output', - side_effect=CalledProcessError(*(None,) * 4), - ): - assert docker.docker_is_running() is False +DOCKER_CGROUPS_V2_MOUNTINFO_EXAMPLE = b'''\ +721 386 0:45 / / rw,relatime master:218 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/QHZ7OM7P4AQD3XLG274ZPWAJCV:/var/lib/docker/overlay2/l/5RFG6SZWVGOG2NKEYXJDQCQYX5,upperdir=/var/lib/docker/overlay2/e4ad859fc5d4791932b9b976052f01fb0063e01de3cef916e40ae2121f6a166e/diff,workdir=/var/lib/docker/overlay2/e4ad859fc5d4791932b9b976052f01fb0063e01de3cef916e40ae2121f6a166e/work,nouserxattr +722 721 0:48 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw +723 721 0:50 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +724 723 0:51 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666 +725 721 0:52 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro +726 725 0:26 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate,memory_recursiveprot +727 723 0:47 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw +728 723 0:53 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k,inode64 +729 721 8:3 /var/lib/docker/containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/sda3 rw,errors=remount-ro +730 721 8:3 /var/lib/docker/containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/hostname /etc/hostname rw,relatime - ext4 /dev/sda3 rw,errors=remount-ro +731 721 8:3 /var/lib/docker/containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/hosts /etc/hosts rw,relatime - ext4 /dev/sda3 rw,errors=remount-ro +387 723 0:51 /0 /dev/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666 +388 722 0:48 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw +389 722 0:48 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw +525 722 0:48 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw +526 722 0:48 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw +571 722 0:48 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw +572 722 0:57 / /proc/asound ro,relatime - tmpfs tmpfs ro,inode64 +575 722 0:58 / /proc/acpi ro,relatime - tmpfs tmpfs ro,inode64 +576 722 0:50 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +577 722 0:50 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +578 722 0:50 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64 +579 722 0:59 / /proc/scsi ro,relatime - tmpfs tmpfs ro,inode64 +580 725 0:60 / /sys/firmware ro,relatime - tmpfs tmpfs ro,inode64 +''' # noqa: E501 + +PODMAN_CGROUPS_V1_MOUNTINFO_EXAMPLE = b'''\ +1200 915 0:57 / / rw,relatime - overlay overlay rw,lowerdir=/home/asottile/.local/share/containers/storage/overlay/l/ZWAU3VY3ZHABQJRBUAFPBX7R5D,upperdir=/home/asottile/.local/share/containers/storage/overlay/72504ef163fda63838930450553b7306412ccad139a007626732b3dc43af5200/diff,workdir=/home/asottile/.local/share/containers/storage/overlay/72504ef163fda63838930450553b7306412ccad139a007626732b3dc43af5200/work,volatile,userxattr +1204 1200 0:62 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw +1205 1200 0:63 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,uid=1000,gid=1000,inode64 +1206 1200 0:64 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw +1207 1205 0:65 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=100004,mode=620,ptmxmode=666 +1208 1205 0:61 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw +1209 1200 0:53 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/.containerenv /run/.containerenv rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=814036k,mode=700,uid=1000,gid=1000,inode64 +1210 1200 0:53 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/resolv.conf /etc/resolv.conf rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=814036k,mode=700,uid=1000,gid=1000,inode64 +1211 1200 0:53 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/hosts /etc/hosts rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=814036k,mode=700,uid=1000,gid=1000,inode64 +1212 1205 0:56 / /dev/shm rw,relatime - tmpfs shm rw,size=64000k,uid=1000,gid=1000,inode64 +1213 1200 0:53 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/hostname /etc/hostname rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=814036k,mode=700,uid=1000,gid=1000,inode64 +1214 1206 0:66 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs cgroup rw,size=1024k,uid=1000,gid=1000,inode64 +1215 1214 0:43 / /sys/fs/cgroup/freezer ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,freezer +1216 1214 0:42 /user.slice /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices +1217 1214 0:41 / /sys/fs/cgroup/hugetlb ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,hugetlb +1218 1214 0:40 / /sys/fs/cgroup/misc ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,misc +1219 1214 0:39 / /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio +1220 1214 0:38 / /sys/fs/cgroup/net_cls,net_prio ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,net_cls,net_prio +1221 1214 0:37 / /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,perf_event +1222 1214 0:36 /user.slice/user-1000.slice/user@1000.service /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory +1223 1214 0:35 /user.slice/user-1000.slice/user@1000.service /sys/fs/cgroup/pids ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids +1224 1214 0:34 / /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpuset +1225 1214 0:33 / /sys/fs/cgroup/cpu,cpuacct ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpu,cpuacct +1226 1214 0:32 / /sys/fs/cgroup/rdma ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,rdma +1227 1214 0:29 /user.slice/user-1000.slice/user@1000.service/apps.slice/apps-org.gnome.Terminal.slice/vte-spawn-0c50448e-b395-4d76-8b92-379f16e5066f.scope /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,xattr,name=systemd +1228 1205 0:5 /null /dev/null rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1229 1205 0:5 /zero /dev/zero rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1230 1205 0:5 /full /dev/full rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1231 1205 0:5 /tty /dev/tty rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1232 1205 0:5 /random /dev/random rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1233 1205 0:5 /urandom /dev/urandom rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1234 1204 0:67 / /proc/acpi ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +1235 1204 0:5 /null /proc/kcore rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1236 1204 0:5 /null /proc/keys rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1237 1204 0:5 /null /proc/timer_list rw,nosuid,noexec,relatime - devtmpfs udev rw,size=4031656k,nr_inodes=1007914,mode=755,inode64 +1238 1204 0:68 / /proc/scsi ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +1239 1206 0:69 / /sys/firmware ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +1240 1206 0:70 / /sys/dev/block ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +1241 1204 0:62 /asound /proc/asound ro,relatime - proc proc rw +1242 1204 0:62 /bus /proc/bus ro,relatime - proc proc rw +1243 1204 0:62 /fs /proc/fs ro,relatime - proc proc rw +1244 1204 0:62 /irq /proc/irq ro,relatime - proc proc rw +1245 1204 0:62 /sys /proc/sys ro,relatime - proc proc rw +1256 1204 0:62 /sysrq-trigger /proc/sysrq-trigger ro,relatime - proc proc rw +916 1205 0:65 /0 /dev/console rw,relatime - devpts devpts rw,gid=100004,mode=620,ptmxmode=666 +''' # noqa: E501 + +PODMAN_CGROUPS_V2_MOUNTINFO_EXAMPLE = b'''\ +685 690 0:63 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/resolv.conf /etc/resolv.conf rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=1637624k,nr_inodes=409406,mode=700,uid=1000,gid=1000,inode64 +686 690 0:63 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/hosts /etc/hosts rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=1637624k,nr_inodes=409406,mode=700,uid=1000,gid=1000,inode64 +687 692 0:50 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=64000k,uid=1000,gid=1000,inode64 +688 690 0:63 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/.containerenv /run/.containerenv rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=1637624k,nr_inodes=409406,mode=700,uid=1000,gid=1000,inode64 +689 690 0:63 /containers/overlay-containers/c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7/userdata/hostname /etc/hostname rw,nosuid,nodev,relatime - tmpfs tmpfs rw,size=1637624k,nr_inodes=409406,mode=700,uid=1000,gid=1000,inode64 +690 546 0:55 / / rw,relatime - overlay overlay rw,lowerdir=/home/asottile/.local/share/containers/storage/overlay/l/NPOHYOD3PI3YW6TQSGBOVOUSK6,upperdir=/home/asottile/.local/share/containers/storage/overlay/565c206fb79f876ffd5f069b8bd7a97fb5e47d5d07396b0c395a4ed6725d4a8e/diff,workdir=/home/asottile/.local/share/containers/storage/overlay/565c206fb79f876ffd5f069b8bd7a97fb5e47d5d07396b0c395a4ed6725d4a8e/work,redirect_dir=nofollow,uuid=on,volatile,userxattr +691 690 0:59 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw +692 690 0:61 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,uid=1000,gid=1000,inode64 +693 690 0:62 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw +694 692 0:66 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=100004,mode=620,ptmxmode=666 +695 692 0:58 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw +696 693 0:28 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot +698 692 0:6 /null /dev/null rw,nosuid,noexec,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +699 692 0:6 /zero /dev/zero rw,nosuid,noexec,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +700 692 0:6 /full /dev/full rw,nosuid,noexec,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +701 692 0:6 /tty /dev/tty rw,nosuid,noexec,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +702 692 0:6 /random /dev/random rw,nosuid,noexec,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +703 692 0:6 /urandom /dev/urandom rw,nosuid,noexec,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +704 691 0:67 / /proc/acpi ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +705 691 0:6 /null /proc/kcore ro,nosuid,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +706 691 0:6 /null /proc/keys ro,nosuid,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +707 691 0:6 /null /proc/latency_stats ro,nosuid,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +708 691 0:6 /null /proc/timer_list ro,nosuid,relatime - devtmpfs udev rw,size=8147812k,nr_inodes=2036953,mode=755,inode64 +709 691 0:68 / /proc/scsi ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +710 693 0:69 / /sys/firmware ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +711 693 0:70 / /sys/dev/block ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +712 693 0:71 / /sys/devices/virtual/powercap ro,relatime - tmpfs tmpfs rw,size=0k,uid=1000,gid=1000,inode64 +713 691 0:59 /asound /proc/asound ro,nosuid,nodev,noexec,relatime - proc proc rw +714 691 0:59 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw +715 691 0:59 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw +716 691 0:59 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw +717 691 0:59 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw +718 691 0:59 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw +547 692 0:66 /0 /dev/console rw,relatime - devpts devpts rw,gid=100004,mode=620,ptmxmode=666 +''' # noqa: E501 + +# The ID should match the above cgroup example. +CONTAINER_ID = 'c33988ec7651ebc867cb24755eaf637a6734088bc7eef59d5799293a9e5450f7' # noqa: E501 + +NON_DOCKER_MOUNTINFO_EXAMPLE = b'''\ +21 27 0:19 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw +22 27 0:20 / /proc rw,nosuid,nodev,noexec,relatime shared:14 - proc proc rw +23 27 0:5 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=10219484k,nr_inodes=2554871,mode=755,inode64 +24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 +25 27 0:22 / /run rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=2047768k,mode=755,inode64 +27 1 8:2 / / rw,relatime shared:1 - ext4 /dev/sda2 rw,errors=remount-ro +28 21 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw +29 23 0:24 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64 +30 25 0:25 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k,inode64 +''' # noqa: E501 def test_docker_fallback_user(): def invalid_attribute(): raise AttributeError + with mock.patch.multiple( - 'os', create=True, - getuid=invalid_attribute, - getgid=invalid_attribute, + 'os', create=True, + getuid=invalid_attribute, + getgid=invalid_attribute, ): - assert docker.get_docker_user() == '1000:1000' + assert docker.get_docker_user() == () + + +@pytest.fixture(autouse=True) +def _avoid_cache(): + with mock.patch.object( + docker, + '_is_rootless', + docker._is_rootless.__wrapped__, + ): + yield + + +@pytest.mark.parametrize( + 'info_ret', + ( + (0, b'{"SecurityOptions": ["name=rootless","name=cgroupns"]}', b''), + (0, b'{"host": {"security": {"rootless": true}}}', b''), + ), +) +def test_docker_user_rootless(info_ret): + with mock.patch.object(docker, 'cmd_output_b', return_value=info_ret): + assert docker.get_docker_user() == () + + +@pytest.mark.parametrize( + 'info_ret', + ( + (0, b'{"SecurityOptions": ["name=cgroupns"]}', b''), + (0, b'{"host": {"security": {"rootless": false}}}', b''), + (0, b'{"response_from_some_other_container_engine": true}', b''), + (0, b'{"SecurityOptions": null}', b''), + (1, b'', b''), + ), +) +def test_docker_user_non_rootless(info_ret): + with mock.patch.object(docker, 'cmd_output_b', return_value=info_ret): + assert docker.get_docker_user() != () + + +def test_container_id_no_file(): + with mock.patch.object(builtins, 'open', side_effect=FileNotFoundError): + assert docker._get_container_id() is None + + +def _mock_open(data): + return mock.patch.object( + builtins, + 'open', + new_callable=mock.mock_open, + read_data=data, + ) + + +def test_container_id_not_in_file(): + with _mock_open(NON_DOCKER_MOUNTINFO_EXAMPLE): + assert docker._get_container_id() is None + + +def test_get_container_id(): + with _mock_open(DOCKER_CGROUPS_V1_MOUNTINFO_EXAMPLE): + assert docker._get_container_id() == CONTAINER_ID + with _mock_open(DOCKER_CGROUPS_V2_MOUNTINFO_EXAMPLE): + assert docker._get_container_id() == CONTAINER_ID + with _mock_open(PODMAN_CGROUPS_V1_MOUNTINFO_EXAMPLE): + assert docker._get_container_id() == CONTAINER_ID + with _mock_open(PODMAN_CGROUPS_V2_MOUNTINFO_EXAMPLE): + assert docker._get_container_id() == CONTAINER_ID + + +def test_get_docker_path_not_in_docker_returns_same(): + with _mock_open(b''): + assert docker._get_docker_path('abc') == 'abc' + + +@pytest.fixture +def in_docker(): + with mock.patch.object( + docker, '_get_container_id', return_value=CONTAINER_ID, + ): + yield + + +def _linux_commonpath(): + return mock.patch.object(os.path, 'commonpath', posixpath.commonpath) + + +def _nt_commonpath(): + return mock.patch.object(os.path, 'commonpath', ntpath.commonpath) + + +def _docker_output(out): + ret = (0, out, b'') + return mock.patch.object(docker, 'cmd_output_b', return_value=ret) + + +def test_get_docker_path_in_docker_no_binds_same_path(in_docker): + docker_out = json.dumps([{'Mounts': []}]).encode() + + with _docker_output(docker_out): + assert docker._get_docker_path('abc') == 'abc' + + +def test_get_docker_path_in_docker_binds_path_equal(in_docker): + binds_list = [{'Source': '/opt/my_code', 'Destination': '/project'}] + docker_out = json.dumps([{'Mounts': binds_list}]).encode() + + with _linux_commonpath(), _docker_output(docker_out): + assert docker._get_docker_path('/project') == '/opt/my_code' + + +def test_get_docker_path_in_docker_binds_path_complex(in_docker): + binds_list = [{'Source': '/opt/my_code', 'Destination': '/project'}] + docker_out = json.dumps([{'Mounts': binds_list}]).encode() + + with _linux_commonpath(), _docker_output(docker_out): + path = '/project/test/something' + assert docker._get_docker_path(path) == '/opt/my_code/test/something' + + +def test_get_docker_path_in_docker_no_substring(in_docker): + binds_list = [{'Source': '/opt/my_code', 'Destination': '/project'}] + docker_out = json.dumps([{'Mounts': binds_list}]).encode() + + with _linux_commonpath(), _docker_output(docker_out): + path = '/projectSuffix/test/something' + assert docker._get_docker_path(path) == path + + +def test_get_docker_path_in_docker_binds_path_many_binds(in_docker): + binds_list = [ + {'Source': '/something_random', 'Destination': '/not-related'}, + {'Source': '/opt/my_code', 'Destination': '/project'}, + {'Source': '/something-random-2', 'Destination': '/not-related-2'}, + ] + docker_out = json.dumps([{'Mounts': binds_list}]).encode() + + with _linux_commonpath(), _docker_output(docker_out): + assert docker._get_docker_path('/project') == '/opt/my_code' + + +def test_get_docker_path_in_docker_windows(in_docker): + binds_list = [{'Source': r'c:\users\user', 'Destination': r'c:\folder'}] + docker_out = json.dumps([{'Mounts': binds_list}]).encode() + + with _nt_commonpath(), _docker_output(docker_out): + path = r'c:\folder\test\something' + expected = r'c:\users\user\test\something' + assert docker._get_docker_path(path) == expected + + +def test_get_docker_path_in_docker_docker_in_docker(in_docker): + # won't be able to discover "self" container in true docker-in-docker + err = CalledProcessError(1, (), b'', b'') + with mock.patch.object(docker, 'cmd_output_b', side_effect=err): + assert docker._get_docker_path('/project') == '/project' + + +@xfailif_windows # pragma: win32 no cover +def test_docker_hook(tmp_path): + dockerfile = '''\ +FROM ubuntu:22.04 +CMD ["echo", "This is overwritten by the entry"'] +''' + tmp_path.joinpath('Dockerfile').write_text(dockerfile) + + ret = run_language(tmp_path, docker, 'echo hello hello world') + assert ret == (0, b'hello hello world\n') + + +@xfailif_windows # pragma: win32 no cover +def test_docker_hook_mount_permissions(tmp_path): + dockerfile = '''\ +FROM ubuntu:22.04 +''' + tmp_path.joinpath('Dockerfile').write_text(dockerfile) + + retcode, _ = run_language(tmp_path, docker, 'touch', ('README.md',)) + assert retcode == 0 diff --git a/tests/languages/dotnet_test.py b/tests/languages/dotnet_test.py new file mode 100644 index 00000000..ee408256 --- /dev/null +++ b/tests/languages/dotnet_test.py @@ -0,0 +1,154 @@ +from __future__ import annotations + +from pre_commit.languages import dotnet +from testing.language_helpers import run_language + + +def _write_program_cs(tmp_path): + program_cs = '''\ +using System; + +namespace dotnet_tests +{ + class Program + { + static void Main(string[] args) + { + Console.WriteLine("Hello from dotnet!"); + } + } +} +''' + tmp_path.joinpath('Program.cs').write_text(program_cs) + + +def _csproj(tool_name): + return f'''\ + + + Exe + net8 + true + {tool_name} + ./nupkg + + +''' + + +def test_dotnet_csproj(tmp_path): + csproj = _csproj('testeroni') + _write_program_cs(tmp_path) + tmp_path.joinpath('dotnet_csproj.csproj').write_text(csproj) + ret = run_language(tmp_path, dotnet, 'testeroni') + assert ret == (0, b'Hello from dotnet!\n') + + +def test_dotnet_csproj_prefix(tmp_path): + csproj = _csproj('testeroni.tool') + _write_program_cs(tmp_path) + tmp_path.joinpath('dotnet_hooks_csproj_prefix.csproj').write_text(csproj) + ret = run_language(tmp_path, dotnet, 'testeroni.tool') + assert ret == (0, b'Hello from dotnet!\n') + + +def test_dotnet_sln(tmp_path): + csproj = _csproj('testeroni') + sln = '''\ +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio 15 +VisualStudioVersion = 15.0.26124.0 +MinimumVisualStudioVersion = 15.0.26124.0 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "dotnet_hooks_sln_repo", "dotnet_hooks_sln_repo.csproj", "{6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Debug|x64 = Debug|x64 + Debug|x86 = Debug|x86 + Release|Any CPU = Release|Any CPU + Release|x64 = Release|x64 + Release|x86 = Release|x86 + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Debug|Any CPU.Build.0 = Debug|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Debug|x64.ActiveCfg = Debug|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Debug|x64.Build.0 = Debug|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Debug|x86.ActiveCfg = Debug|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Debug|x86.Build.0 = Debug|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Release|Any CPU.ActiveCfg = Release|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Release|Any CPU.Build.0 = Release|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Release|x64.ActiveCfg = Release|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Release|x64.Build.0 = Release|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Release|x86.ActiveCfg = Release|Any CPU + {6568CFDB-6F6F-45A9-932C-8C7DAABC8E56}.Release|x86.Build.0 = Release|Any CPU + EndGlobalSection +EndGlobal +''' # noqa: E501 + _write_program_cs(tmp_path) + tmp_path.joinpath('dotnet_hooks_sln_repo.csproj').write_text(csproj) + tmp_path.joinpath('dotnet_hooks_sln_repo.sln').write_text(sln) + + ret = run_language(tmp_path, dotnet, 'testeroni') + assert ret == (0, b'Hello from dotnet!\n') + + +def _setup_dotnet_combo(tmp_path): + sln = '''\ +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio Version 16 +VisualStudioVersion = 16.0.30114.105 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "proj1", "proj1\\proj1.csproj", "{38A939C3-DEA4-47D7-9B75-0418C4249662}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "proj2", "proj2\\proj2.csproj", "{4C9916CB-165C-4EF5-8A57-4CB6794C1EBF}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {38A939C3-DEA4-47D7-9B75-0418C4249662}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {38A939C3-DEA4-47D7-9B75-0418C4249662}.Debug|Any CPU.Build.0 = Debug|Any CPU + {38A939C3-DEA4-47D7-9B75-0418C4249662}.Release|Any CPU.ActiveCfg = Release|Any CPU + {38A939C3-DEA4-47D7-9B75-0418C4249662}.Release|Any CPU.Build.0 = Release|Any CPU + {4C9916CB-165C-4EF5-8A57-4CB6794C1EBF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {4C9916CB-165C-4EF5-8A57-4CB6794C1EBF}.Debug|Any CPU.Build.0 = Debug|Any CPU + {4C9916CB-165C-4EF5-8A57-4CB6794C1EBF}.Release|Any CPU.ActiveCfg = Release|Any CPU + {4C9916CB-165C-4EF5-8A57-4CB6794C1EBF}.Release|Any CPU.Build.0 = Release|Any CPU + EndGlobalSection +EndGlobal +''' # noqa: E501 + tmp_path.joinpath('dotnet_hooks_combo_repo.sln').write_text(sln) + + csproj1 = _csproj('proj1') + proj1 = tmp_path.joinpath('proj1') + proj1.mkdir() + proj1.joinpath('proj1.csproj').write_text(csproj1) + _write_program_cs(proj1) + + csproj2 = _csproj('proj2') + proj2 = tmp_path.joinpath('proj2') + proj2.mkdir() + proj2.joinpath('proj2.csproj').write_text(csproj2) + _write_program_cs(proj2) + + +def test_dotnet_combo_proj1(tmp_path): + _setup_dotnet_combo(tmp_path) + ret = run_language(tmp_path, dotnet, 'proj1') + assert ret == (0, b'Hello from dotnet!\n') + + +def test_dotnet_combo_proj2(tmp_path): + _setup_dotnet_combo(tmp_path) + ret = run_language(tmp_path, dotnet, 'proj2') + assert ret == (0, b'Hello from dotnet!\n') diff --git a/tests/languages/fail_test.py b/tests/languages/fail_test.py new file mode 100644 index 00000000..7c74886f --- /dev/null +++ b/tests/languages/fail_test.py @@ -0,0 +1,14 @@ +from __future__ import annotations + +from pre_commit.languages import fail +from testing.language_helpers import run_language + + +def test_fail_hooks(tmp_path): + ret = run_language( + tmp_path, + fail, + 'watch out for', + file_args=('bunnies',), + ) + assert ret == (1, b'watch out for\n\nbunnies\n') diff --git a/tests/languages/golang_test.py b/tests/languages/golang_test.py index 483f41ea..7fb6ab18 100644 --- a/tests/languages/golang_test.py +++ b/tests/languages/golang_test.py @@ -1,23 +1,236 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations + +from unittest import mock import pytest +import re_assert -from pre_commit.languages.golang import guess_go_dir +import pre_commit.constants as C +from pre_commit import lang_base +from pre_commit.commands.install_uninstall import install +from pre_commit.envcontext import envcontext +from pre_commit.languages import golang +from pre_commit.store import _make_local_repo +from pre_commit.util import CalledProcessError +from pre_commit.util import cmd_output +from testing.fixtures import add_config_to_repo +from testing.fixtures import make_config_from_repo +from testing.language_helpers import run_language +from testing.util import cmd_output_mocked_pre_commit_home +from testing.util import cwd +from testing.util import git_commit -@pytest.mark.parametrize( - ('url', 'expected'), - ( - ('/im/a/path/on/disk', 'unknown_src_dir'), - ('file:///im/a/path/on/disk', 'unknown_src_dir'), - ('git@github.com:golang/lint', 'github.com/golang/lint'), - ('git://github.com/golang/lint', 'github.com/golang/lint'), - ('http://github.com/golang/lint', 'github.com/golang/lint'), - ('https://github.com/golang/lint', 'github.com/golang/lint'), - ('ssh://git@github.com/golang/lint', 'github.com/golang/lint'), - ('git@github.com:golang/lint.git', 'github.com/golang/lint'), - ), +ACTUAL_GET_DEFAULT_VERSION = golang.get_default_version.__wrapped__ + + +@pytest.fixture +def exe_exists_mck(): + with mock.patch.object(lang_base, 'exe_exists') as mck: + yield mck + + +def test_golang_default_version_system_available(exe_exists_mck): + exe_exists_mck.return_value = True + assert ACTUAL_GET_DEFAULT_VERSION() == 'system' + + +def test_golang_default_version_system_not_available(exe_exists_mck): + exe_exists_mck.return_value = False + assert ACTUAL_GET_DEFAULT_VERSION() == C.DEFAULT + + +ACTUAL_INFER_GO_VERSION = golang._infer_go_version.__wrapped__ + + +def test_golang_infer_go_version_not_default(): + assert ACTUAL_INFER_GO_VERSION('1.19.4') == '1.19.4' + + +def test_golang_infer_go_version_default(): + version = ACTUAL_INFER_GO_VERSION(C.DEFAULT) + + assert version != C.DEFAULT + re_assert.Matches(r'^\d+\.\d+(?:\.\d+)?$').assert_matches(version) + + +def _make_hello_world(tmp_path): + go_mod = '''\ +module golang-hello-world + +go 1.18 + +require github.com/BurntSushi/toml v1.1.0 +''' + go_sum = '''\ +github.com/BurntSushi/toml v1.1.0 h1:ksErzDEI1khOiGPgpwuI7x2ebx/uXQNw7xJpn9Eq1+I= +github.com/BurntSushi/toml v1.1.0/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +''' # noqa: E501 + hello_world_go = '''\ +package main + + +import ( + "fmt" + "github.com/BurntSushi/toml" ) -def test_guess_go_dir(url, expected): - assert guess_go_dir(url) == expected + +type Config struct { + What string +} + +func main() { + var conf Config + toml.Decode("What = 'world'\\n", &conf) + fmt.Printf("hello %v\\n", conf.What) +} +''' + tmp_path.joinpath('go.mod').write_text(go_mod) + tmp_path.joinpath('go.sum').write_text(go_sum) + mod_dir = tmp_path.joinpath('golang-hello-world') + mod_dir.mkdir() + main_file = mod_dir.joinpath('main.go') + main_file.write_text(hello_world_go) + + +def test_golang_system(tmp_path): + _make_hello_world(tmp_path) + + ret = run_language(tmp_path, golang, 'golang-hello-world') + assert ret == (0, b'hello world\n') + + +def test_golang_default_version(tmp_path): + _make_hello_world(tmp_path) + + ret = run_language( + tmp_path, + golang, + 'golang-hello-world', + version=C.DEFAULT, + ) + assert ret == (0, b'hello world\n') + + +def test_golang_versioned(tmp_path): + _make_local_repo(str(tmp_path)) + + ret, out = run_language( + tmp_path, + golang, + 'go version', + version='1.21.1', + ) + + assert ret == 0 + assert out.startswith(b'go version go1.21.1') + + +def test_local_golang_additional_deps(tmp_path): + _make_local_repo(str(tmp_path)) + + ret = run_language( + tmp_path, + golang, + 'hello', + deps=('golang.org/x/example/hello@latest',), + ) + + assert ret == (0, b'Hello, world!\n') + + +def test_golang_hook_still_works_when_gobin_is_set(tmp_path): + with envcontext((('GOBIN', str(tmp_path.joinpath('gobin'))),)): + test_golang_system(tmp_path) + + +def test_during_commit_all(tmp_path, tempdir_factory, store, in_git_dir): + hook_dir = tmp_path.joinpath('hook') + hook_dir.mkdir() + _make_hello_world(hook_dir) + hook_dir.joinpath('.pre-commit-hooks.yaml').write_text( + '- id: hello-world\n' + ' name: hello world\n' + ' entry: golang-hello-world\n' + ' language: golang\n' + ' always_run: true\n', + ) + cmd_output('git', 'init', hook_dir) + cmd_output('git', 'add', '.', cwd=hook_dir) + git_commit(cwd=hook_dir) + + add_config_to_repo(in_git_dir, make_config_from_repo(hook_dir)) + + assert not install(C.CONFIG_FILE, store, hook_types=['pre-commit']) + + git_commit( + fn=cmd_output_mocked_pre_commit_home, + tempdir_factory=tempdir_factory, + ) + + +def test_automatic_toolchain_switching(tmp_path): + go_mod = '''\ +module toolchain-version-test + +go 1.23.1 +''' + main_go = '''\ +package main + +func main() {} +''' + tmp_path.joinpath('go.mod').write_text(go_mod) + mod_dir = tmp_path.joinpath('toolchain-version-test') + mod_dir.mkdir() + main_file = mod_dir.joinpath('main.go') + main_file.write_text(main_go) + + with pytest.raises(CalledProcessError) as excinfo: + run_language( + path=tmp_path, + language=golang, + version='1.22.0', + exe='golang-version-test', + ) + + assert 'go.mod requires go >= 1.23.1' in excinfo.value.stderr.decode() + + +def test_automatic_toolchain_switching_go_fmt(tmp_path, monkeypatch): + go_mod_hook = '''\ +module toolchain-version-test + +go 1.22.0 +''' + go_mod = '''\ +module toolchain-version-test + +go 1.23.1 +''' + main_go = '''\ +package main + +func main() {} +''' + hook_dir = tmp_path.joinpath('hook') + hook_dir.mkdir() + hook_dir.joinpath('go.mod').write_text(go_mod_hook) + + test_dir = tmp_path.joinpath('test') + test_dir.mkdir() + test_dir.joinpath('go.mod').write_text(go_mod) + main_file = test_dir.joinpath('main.go') + main_file.write_text(main_go) + + with cwd(test_dir): + ret, out = run_language( + path=hook_dir, + language=golang, + version='1.22.0', + exe='go fmt', + file_args=(str(main_file),), + ) + + assert ret == 1 + assert 'go.mod requires go >= 1.23.1' in out.decode() diff --git a/tests/languages/haskell_test.py b/tests/languages/haskell_test.py new file mode 100644 index 00000000..f888109b --- /dev/null +++ b/tests/languages/haskell_test.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +import pytest + +from pre_commit.errors import FatalError +from pre_commit.languages import haskell +from pre_commit.util import win_exe +from testing.language_helpers import run_language + + +def test_run_example_executable(tmp_path): + example_cabal = '''\ +cabal-version: 2.4 +name: example +version: 0.1.0.0 + +executable example + main-is: Main.hs + + build-depends: base >=4 + default-language: Haskell2010 +''' + main_hs = '''\ +module Main where + +main :: IO () +main = putStrLn "Hello, Haskell!" +''' + tmp_path.joinpath('example.cabal').write_text(example_cabal) + tmp_path.joinpath('Main.hs').write_text(main_hs) + + result = run_language(tmp_path, haskell, 'example') + assert result == (0, b'Hello, Haskell!\n') + + # should not symlink things into environments + exe = tmp_path.joinpath(win_exe('hs_env-default/bin/example')) + assert exe.is_file() + assert not exe.is_symlink() + + +def test_run_dep(tmp_path): + result = run_language(tmp_path, haskell, 'hello', deps=['hello']) + assert result == (0, b'Hello, World!\n') + + +def test_run_empty(tmp_path): + with pytest.raises(FatalError) as excinfo: + run_language(tmp_path, haskell, 'example') + msg, = excinfo.value.args + assert msg == 'Expected .cabal files or additional_dependencies' diff --git a/tests/languages/helpers_test.py b/tests/languages/helpers_test.py deleted file mode 100644 index 629322c3..00000000 --- a/tests/languages/helpers_test.py +++ /dev/null @@ -1,83 +0,0 @@ -from __future__ import absolute_import -from __future__ import unicode_literals - -import multiprocessing -import os -import sys - -import mock -import pytest - -import pre_commit.constants as C -from pre_commit.languages import helpers -from pre_commit.prefix import Prefix -from pre_commit.util import CalledProcessError -from testing.auto_namedtuple import auto_namedtuple - - -def test_basic_get_default_version(): - assert helpers.basic_get_default_version() == C.DEFAULT - - -def test_basic_healthy(): - assert helpers.basic_healthy(None, None) is True - - -def test_failed_setup_command_does_not_unicode_error(): - script = ( - 'import sys\n' - "getattr(sys.stderr, 'buffer', sys.stderr).write(b'\\x81\\xfe')\n" - 'exit(1)\n' - ) - - # an assertion that this does not raise `UnicodeError` - with pytest.raises(CalledProcessError): - helpers.run_setup_cmd(Prefix('.'), (sys.executable, '-c', script)) - - -def test_assert_no_additional_deps(): - with pytest.raises(AssertionError) as excinfo: - helpers.assert_no_additional_deps('lang', ['hmmm']) - msg, = excinfo.value.args - assert msg == ( - 'For now, pre-commit does not support additional_dependencies for lang' - ) - - -SERIAL_FALSE = auto_namedtuple(require_serial=False) -SERIAL_TRUE = auto_namedtuple(require_serial=True) - - -def test_target_concurrency_normal(): - with mock.patch.object(multiprocessing, 'cpu_count', return_value=123): - with mock.patch.dict(os.environ, {}, clear=True): - assert helpers.target_concurrency(SERIAL_FALSE) == 123 - - -def test_target_concurrency_cpu_count_require_serial_true(): - with mock.patch.dict(os.environ, {}, clear=True): - assert helpers.target_concurrency(SERIAL_TRUE) == 1 - - -def test_target_concurrency_testing_env_var(): - with mock.patch.dict( - os.environ, {'PRE_COMMIT_NO_CONCURRENCY': '1'}, clear=True, - ): - assert helpers.target_concurrency(SERIAL_FALSE) == 1 - - -def test_target_concurrency_on_travis(): - with mock.patch.dict(os.environ, {'TRAVIS': '1'}, clear=True): - assert helpers.target_concurrency(SERIAL_FALSE) == 2 - - -def test_target_concurrency_cpu_count_not_implemented(): - with mock.patch.object( - multiprocessing, 'cpu_count', side_effect=NotImplementedError, - ): - with mock.patch.dict(os.environ, {}, clear=True): - assert helpers.target_concurrency(SERIAL_FALSE) == 1 - - -def test_shuffled_is_deterministic(): - assert helpers._shuffled(range(10)) == [3, 7, 8, 2, 4, 6, 5, 1, 0, 9] diff --git a/tests/languages/julia_test.py b/tests/languages/julia_test.py new file mode 100644 index 00000000..175622d6 --- /dev/null +++ b/tests/languages/julia_test.py @@ -0,0 +1,111 @@ +from __future__ import annotations + +import os +from unittest import mock + +from pre_commit.languages import julia +from testing.language_helpers import run_language +from testing.util import cwd + + +def _make_hook(tmp_path, julia_code): + src_dir = tmp_path.joinpath('src') + src_dir.mkdir() + src_dir.joinpath('main.jl').write_text(julia_code) + tmp_path.joinpath('Project.toml').write_text( + '[deps]\n' + 'Example = "7876af07-990d-54b4-ab0e-23690620f79a"\n', + ) + + +def test_julia_hook(tmp_path): + code = """ + using Example + function main() + println("Hello, world!") + end + main() + """ + _make_hook(tmp_path, code) + expected = (0, b'Hello, world!\n') + assert run_language(tmp_path, julia, 'src/main.jl') == expected + + +def test_julia_hook_with_startup(tmp_path): + depot_path = tmp_path.joinpath('depot') + depot_path.joinpath('config').mkdir(parents=True) + startup = depot_path.joinpath('config', 'startup.jl') + startup.write_text('error("Startup file used!")\n') + + depo_path_var = f'{depot_path}{os.pathsep}' + with mock.patch.dict(os.environ, {'JULIA_DEPOT_PATH': depo_path_var}): + test_julia_hook(tmp_path) + + +def test_julia_hook_manifest(tmp_path): + code = """ + using Example + println(pkgversion(Example)) + """ + _make_hook(tmp_path, code) + + tmp_path.joinpath('Manifest.toml').write_text( + 'manifest_format = "2.0"\n\n' + '[[deps.Example]]\n' + 'git-tree-sha1 = "11820aa9c229fd3833d4bd69e5e75ef4e7273bf1"\n' + 'uuid = "7876af07-990d-54b4-ab0e-23690620f79a"\n' + 'version = "0.5.4"\n', + ) + expected = (0, b'0.5.4\n') + assert run_language(tmp_path, julia, 'src/main.jl') == expected + + +def test_julia_hook_args(tmp_path): + code = """ + function main(argv) + foreach(println, argv) + end + main(ARGS) + """ + _make_hook(tmp_path, code) + expected = (0, b'--arg1\n--arg2\n') + assert run_language( + tmp_path, julia, 'src/main.jl --arg1 --arg2', + ) == expected + + +def test_julia_hook_additional_deps(tmp_path): + code = """ + using TOML + function main() + project_file = Base.active_project() + dict = TOML.parsefile(project_file) + for (k, v) in dict["deps"] + println(k, " = ", v) + end + end + main() + """ + _make_hook(tmp_path, code) + deps = ('TOML=fa267f1f-6049-4f14-aa54-33bafae1ed76',) + ret, out = run_language(tmp_path, julia, 'src/main.jl', deps=deps) + assert ret == 0 + assert b'Example = 7876af07-990d-54b4-ab0e-23690620f79a' in out + assert b'TOML = fa267f1f-6049-4f14-aa54-33bafae1ed76' in out + + +def test_julia_repo_local(tmp_path): + env_dir = tmp_path.joinpath('envdir') + env_dir.mkdir() + local_dir = tmp_path.joinpath('local') + local_dir.mkdir() + local_dir.joinpath('local.jl').write_text( + 'using TOML; foreach(println, ARGS)', + ) + with cwd(local_dir): + deps = ('TOML=fa267f1f-6049-4f14-aa54-33bafae1ed76',) + expected = (0, b'--local-arg1\n--local-arg2\n') + assert run_language( + env_dir, julia, 'local.jl --local-arg1 --local-arg2', + deps=deps, is_local=True, + ) == expected diff --git a/tests/languages/lua_test.py b/tests/languages/lua_test.py new file mode 100644 index 00000000..b2767b72 --- /dev/null +++ b/tests/languages/lua_test.py @@ -0,0 +1,58 @@ +from __future__ import annotations + +import sys + +import pytest + +from pre_commit.languages import lua +from pre_commit.util import make_executable +from testing.language_helpers import run_language + +pytestmark = pytest.mark.skipif( + sys.platform == 'win32', + reason='lua is not supported on windows', +) + + +def test_lua(tmp_path): # pragma: win32 no cover + rockspec = '''\ +package = "hello" +version = "dev-1" + +source = { + url = "git+ssh://git@github.com/pre-commit/pre-commit.git" +} +description = {} +dependencies = {} +build = { + type = "builtin", + modules = {}, + install = { + bin = {"bin/hello-world-lua"} + }, +} +''' + hello_world_lua = '''\ +#!/usr/bin/env lua +print('hello world') +''' + tmp_path.joinpath('hello-dev-1.rockspec').write_text(rockspec) + bin_dir = tmp_path.joinpath('bin') + bin_dir.mkdir() + bin_file = bin_dir.joinpath('hello-world-lua') + bin_file.write_text(hello_world_lua) + make_executable(bin_file) + + expected = (0, b'hello world\n') + assert run_language(tmp_path, lua, 'hello-world-lua') == expected + + +def test_lua_additional_dependencies(tmp_path): # pragma: win32 no cover + ret, out = run_language( + tmp_path, + lua, + 'luacheck --version', + deps=('luacheck',), + ) + assert ret == 0 + assert out.startswith(b'Luacheck: ') diff --git a/tests/languages/node_test.py b/tests/languages/node_test.py new file mode 100644 index 00000000..055cb1e9 --- /dev/null +++ b/tests/languages/node_test.py @@ -0,0 +1,152 @@ +from __future__ import annotations + +import json +import os +import shutil +import sys +from unittest import mock + +import pytest + +import pre_commit.constants as C +from pre_commit import envcontext +from pre_commit import parse_shebang +from pre_commit.languages import node +from pre_commit.prefix import Prefix +from pre_commit.store import _make_local_repo +from pre_commit.util import cmd_output +from testing.language_helpers import run_language +from testing.util import xfailif_windows + + +ACTUAL_GET_DEFAULT_VERSION = node.get_default_version.__wrapped__ + + +@pytest.fixture +def is_linux(): + with mock.patch.object(sys, 'platform', 'linux'): + yield + + +@pytest.fixture +def is_win32(): + with mock.patch.object(sys, 'platform', 'win32'): + yield + + +@pytest.fixture +def find_exe_mck(): + with mock.patch.object(parse_shebang, 'find_executable') as mck: + yield mck + + +@pytest.mark.usefixtures('is_linux') +def test_sets_system_when_node_and_npm_are_available(find_exe_mck): + find_exe_mck.return_value = '/path/to/exe' + assert ACTUAL_GET_DEFAULT_VERSION() == 'system' + + +@pytest.mark.usefixtures('is_linux') +def test_uses_default_when_node_and_npm_are_not_available(find_exe_mck): + find_exe_mck.return_value = None + assert ACTUAL_GET_DEFAULT_VERSION() == C.DEFAULT + + +@pytest.mark.usefixtures('is_win32') +def test_sets_default_on_windows(find_exe_mck): + find_exe_mck.return_value = '/path/to/exe' + assert ACTUAL_GET_DEFAULT_VERSION() == C.DEFAULT + + +@xfailif_windows # pragma: win32 no cover +def test_healthy_system_node(tmpdir): + tmpdir.join('package.json').write('{"name": "t", "version": "1.0.0"}') + + prefix = Prefix(str(tmpdir)) + node.install_environment(prefix, 'system', ()) + assert node.health_check(prefix, 'system') is None + + +@xfailif_windows # pragma: win32 no cover +def test_unhealthy_if_system_node_goes_missing(tmpdir): + bin_dir = tmpdir.join('bin').ensure_dir() + node_bin = bin_dir.join('node') + node_bin.mksymlinkto(shutil.which('node')) + + prefix_dir = tmpdir.join('prefix').ensure_dir() + prefix_dir.join('package.json').write('{"name": "t", "version": "1.0.0"}') + + path = ('PATH', (str(bin_dir), os.pathsep, envcontext.Var('PATH'))) + with envcontext.envcontext((path,)): + prefix = Prefix(str(prefix_dir)) + node.install_environment(prefix, 'system', ()) + assert node.health_check(prefix, 'system') is None + + node_bin.remove() + ret = node.health_check(prefix, 'system') + assert ret == '`node --version` returned 127' + + +@xfailif_windows # pragma: win32 no cover +def test_installs_without_links_outside_env(tmpdir): + tmpdir.join('bin/main.js').ensure().write( + '#!/usr/bin/env node\n' + '_ = require("lodash"); console.log("success!")\n', + ) + tmpdir.join('package.json').write( + json.dumps({ + 'name': 'foo', + 'version': '0.0.1', + 'bin': {'foo': './bin/main.js'}, + 'dependencies': {'lodash': '*'}, + }), + ) + + prefix = Prefix(str(tmpdir)) + node.install_environment(prefix, 'system', ()) + assert node.health_check(prefix, 'system') is None + + # this directory shouldn't exist, make sure we succeed without it existing + cmd_output('rm', '-rf', str(tmpdir.join('node_modules'))) + + with node.in_env(prefix, 'system'): + assert cmd_output('foo')[1] == 'success!\n' + + +def _make_hello_world(tmp_path): + package_json = '''\ +{"name": "t", "version": "0.0.1", "bin": {"node-hello": "./bin/main.js"}} +''' + tmp_path.joinpath('package.json').write_text(package_json) + bin_dir = tmp_path.joinpath('bin') + bin_dir.mkdir() + bin_dir.joinpath('main.js').write_text( + '#!/usr/bin/env node\n' + 'console.log("Hello World");\n', + ) + + +def test_node_hook_system(tmp_path): + _make_hello_world(tmp_path) + ret = run_language(tmp_path, node, 'node-hello') + assert ret == (0, b'Hello World\n') + + +def test_node_with_user_config_set(tmp_path): + cfg = tmp_path.joinpath('cfg') + cfg.write_text('cache=/dne\n') + with envcontext.envcontext((('NPM_CONFIG_USERCONFIG', str(cfg)),)): + test_node_hook_system(tmp_path) + + +@pytest.mark.parametrize('version', (C.DEFAULT, '18.14.0')) +def test_node_hook_versions(tmp_path, version): + _make_hello_world(tmp_path) + ret = run_language(tmp_path, node, 'node-hello', version=version) + assert ret == (0, b'Hello World\n') + + +def test_node_additional_deps(tmp_path): + _make_local_repo(str(tmp_path)) + ret, out = run_language(tmp_path, node, 'npm ls -g', deps=('lodash',)) + assert b' lodash@' in out diff --git a/tests/languages/perl_test.py b/tests/languages/perl_test.py new file mode 100644 index 00000000..042478db --- /dev/null +++ b/tests/languages/perl_test.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +from pre_commit.languages import perl +from pre_commit.store import _make_local_repo +from pre_commit.util import make_executable +from testing.language_helpers import run_language + + +def test_perl_install(tmp_path): + makefile_pl = '''\ +use strict; +use warnings; + +use ExtUtils::MakeMaker; + +WriteMakefile( + NAME => "PreCommitHello", + VERSION_FROM => "lib/PreCommitHello.pm", + EXE_FILES => [qw(bin/pre-commit-perl-hello)], +); +''' + bin_perl_hello = '''\ +#!/usr/bin/env perl + +use strict; +use warnings; +use PreCommitHello; + +PreCommitHello::hello(); +''' + lib_hello_pm = '''\ +package PreCommitHello; + +use strict; +use warnings; + +our $VERSION = "0.1.0"; + +sub hello { + print "Hello from perl-commit Perl!\n"; +} + +1; +''' + tmp_path.joinpath('Makefile.PL').write_text(makefile_pl) + bin_dir = tmp_path.joinpath('bin') + bin_dir.mkdir() + exe = bin_dir.joinpath('pre-commit-perl-hello') + exe.write_text(bin_perl_hello) + make_executable(exe) + lib_dir = tmp_path.joinpath('lib') + lib_dir.mkdir() + lib_dir.joinpath('PreCommitHello.pm').write_text(lib_hello_pm) + + ret = run_language(tmp_path, perl, 'pre-commit-perl-hello') + assert ret == (0, b'Hello from perl-commit Perl!\n') + + +def test_perl_additional_dependencies(tmp_path): + _make_local_repo(str(tmp_path)) + + ret, out = run_language( + tmp_path, + perl, + 'perltidy --version', + deps=('SHANCOCK/Perl-Tidy-20211029.tar.gz',), + ) + assert ret == 0 + assert out.startswith(b'This is perltidy, v20211029') diff --git a/tests/languages/pygrep_test.py b/tests/languages/pygrep_test.py index d91363e2..c6271c80 100644 --- a/tests/languages/pygrep_test.py +++ b/tests/languages/pygrep_test.py @@ -1,9 +1,9 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import pytest from pre_commit.languages import pygrep +from testing.language_helpers import run_language @pytest.fixture @@ -11,6 +11,12 @@ def some_files(tmpdir): tmpdir.join('f1').write_binary(b'foo\nbar\n') tmpdir.join('f2').write_binary(b'[INFO] hi\n') tmpdir.join('f3').write_binary(b"with'quotes\n") + tmpdir.join('f4').write_binary(b'foo\npattern\nbar\n') + tmpdir.join('f5').write_binary(b'[INFO] hi\npattern\nbar') + tmpdir.join('f6').write_binary(b"pattern\nbarwith'foo\n") + tmpdir.join('f7').write_binary(b"hello'hi\nworld\n") + tmpdir.join('f8').write_binary(b'foo\nbar\nbaz\n') + tmpdir.join('f9').write_binary(b'[WARN] hi\n') with tmpdir.as_cwd(): yield @@ -26,43 +32,113 @@ def some_files(tmpdir): ("h'q", 1, "f3:1:with'quotes\n"), ), ) -def test_main(some_files, cap_out, pattern, expected_retcode, expected_out): +def test_main(cap_out, pattern, expected_retcode, expected_out): ret = pygrep.main((pattern, 'f1', 'f2', 'f3')) out = cap_out.get() assert ret == expected_retcode assert out == expected_out -def test_ignore_case(some_files, cap_out): +@pytest.mark.usefixtures('some_files') +def test_negate_by_line_no_match(cap_out): + ret = pygrep.main(('pattern\nbar', 'f4', 'f5', 'f6', '--negate')) + out = cap_out.get() + assert ret == 1 + assert out == 'f4\nf5\nf6\n' + + +@pytest.mark.usefixtures('some_files') +def test_negate_by_line_two_match(cap_out): + ret = pygrep.main(('foo', 'f4', 'f5', 'f6', '--negate')) + out = cap_out.get() + assert ret == 1 + assert out == 'f5\n' + + +@pytest.mark.usefixtures('some_files') +def test_negate_by_line_all_match(cap_out): + ret = pygrep.main(('pattern', 'f4', 'f5', 'f6', '--negate')) + out = cap_out.get() + assert ret == 0 + assert out == '' + + +@pytest.mark.usefixtures('some_files') +def test_negate_by_file_no_match(cap_out): + ret = pygrep.main(('baz', 'f4', 'f5', 'f6', '--negate', '--multiline')) + out = cap_out.get() + assert ret == 1 + assert out == 'f4\nf5\nf6\n' + + +@pytest.mark.usefixtures('some_files') +def test_negate_by_file_one_match(cap_out): + ret = pygrep.main( + ('foo\npattern', 'f4', 'f5', 'f6', '--negate', '--multiline'), + ) + out = cap_out.get() + assert ret == 1 + assert out == 'f5\nf6\n' + + +@pytest.mark.usefixtures('some_files') +def test_negate_by_file_all_match(cap_out): + ret = pygrep.main( + ('pattern\nbar', 'f4', 'f5', 'f6', '--negate', '--multiline'), + ) + out = cap_out.get() + assert ret == 0 + assert out == '' + + +@pytest.mark.usefixtures('some_files') +def test_ignore_case(cap_out): ret = pygrep.main(('--ignore-case', 'info', 'f1', 'f2', 'f3')) out = cap_out.get() assert ret == 1 assert out == 'f2:1:[INFO] hi\n' -def test_multiline(some_files, cap_out): +@pytest.mark.usefixtures('some_files') +def test_multiline(cap_out): ret = pygrep.main(('--multiline', r'foo\nbar', 'f1', 'f2', 'f3')) out = cap_out.get() assert ret == 1 assert out == 'f1:1:foo\nbar\n' -def test_multiline_line_number(some_files, cap_out): +@pytest.mark.usefixtures('some_files') +def test_multiline_line_number(cap_out): ret = pygrep.main(('--multiline', r'ar', 'f1', 'f2', 'f3')) out = cap_out.get() assert ret == 1 assert out == 'f1:2:bar\n' -def test_multiline_dotall_flag_is_enabled(some_files, cap_out): +@pytest.mark.usefixtures('some_files') +def test_multiline_dotall_flag_is_enabled(cap_out): ret = pygrep.main(('--multiline', r'o.*bar', 'f1', 'f2', 'f3')) out = cap_out.get() assert ret == 1 assert out == 'f1:1:foo\nbar\n' -def test_multiline_multiline_flag_is_enabled(some_files, cap_out): +@pytest.mark.usefixtures('some_files') +def test_multiline_multiline_flag_is_enabled(cap_out): ret = pygrep.main(('--multiline', r'foo$.*bar', 'f1', 'f2', 'f3')) out = cap_out.get() assert ret == 1 assert out == 'f1:1:foo\nbar\n' + + +def test_grep_hook_matching(some_files, tmp_path): + ret = run_language( + tmp_path, pygrep, 'ello', file_args=('f7', 'f8', 'f9'), + ) + assert ret == (1, b"f7:1:hello'hi\n") + + +@pytest.mark.parametrize('regex', ('nope', "foo'bar", r'^\[INFO\]')) +def test_grep_hook_not_matching(regex, some_files, tmp_path): + ret = run_language(tmp_path, pygrep, regex, file_args=('f7', 'f8', 'f9')) + assert ret == (0, b'') diff --git a/tests/languages/python_test.py b/tests/languages/python_test.py index 426d3ec6..593634b7 100644 --- a/tests/languages/python_test.py +++ b/tests/languages/python_test.py @@ -1,34 +1,367 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os.path import sys +from unittest import mock -import mock import pytest +import pre_commit.constants as C +from pre_commit.envcontext import envcontext from pre_commit.languages import python +from pre_commit.prefix import Prefix +from pre_commit.store import _make_local_repo +from pre_commit.util import cmd_output_b +from pre_commit.util import make_executable +from pre_commit.util import win_exe +from testing.auto_namedtuple import auto_namedtuple +from testing.language_helpers import run_language + + +def test_read_pyvenv_cfg(tmpdir): + pyvenv_cfg = tmpdir.join('pyvenv.cfg') + pyvenv_cfg.write( + '# I am a comment\n' + '\n' + 'foo = bar\n' + 'version-info=123\n', + ) + expected = {'foo': 'bar', 'version-info': '123'} + assert python._read_pyvenv_cfg(pyvenv_cfg) == expected + + +def test_read_pyvenv_cfg_non_utf8(tmpdir): + pyvenv_cfg = tmpdir.join('pyvenv_cfg') + pyvenv_cfg.write_binary('hello = hello john.Ε‘\n'.encode()) + expected = {'hello': 'hello john.Ε‘'} + assert python._read_pyvenv_cfg(pyvenv_cfg) == expected + + +def _get_default_version( + *, + impl: str, + exe: str, + found: set[str], + version: tuple[int, int], +) -> str: + sys_exe = f'/fake/path/{exe}' + sys_impl = auto_namedtuple(name=impl) + sys_ver = auto_namedtuple(major=version[0], minor=version[1]) + + def find_exe(s): + if s in found: + return f'/fake/path/found/{exe}' + else: + return None + + with ( + mock.patch.object(sys, 'implementation', sys_impl), + mock.patch.object(sys, 'executable', sys_exe), + mock.patch.object(sys, 'version_info', sys_ver), + mock.patch.object(python, 'find_executable', find_exe), + ): + return python.get_default_version.__wrapped__() + + +def test_default_version_sys_executable_found(): + ret = _get_default_version( + impl='cpython', + exe='python3.12', + found={'python3.12'}, + version=(3, 12), + ) + assert ret == 'python3.12' + + +def test_default_version_picks_specific_when_found(): + ret = _get_default_version( + impl='cpython', + exe='python3', + found={'python3', 'python3.12'}, + version=(3, 12), + ) + assert ret == 'python3.12' + + +def test_default_version_picks_pypy_versioned_exe(): + ret = _get_default_version( + impl='pypy', + exe='python', + found={'pypy3.12', 'python3'}, + version=(3, 12), + ) + assert ret == 'pypy3.12' + + +def test_default_version_picks_pypy_unversioned_exe(): + ret = _get_default_version( + impl='pypy', + exe='python', + found={'pypy3', 'python3'}, + version=(3, 12), + ) + assert ret == 'pypy3' def test_norm_version_expanduser(): home = os.path.expanduser('~') - if os.name == 'nt': # pragma: no cover (nt) + if sys.platform == 'win32': # pragma: win32 cover path = r'~\python343' - expected_path = r'{}\python343'.format(home) - else: # pragma: windows no cover + expected_path = fr'{home}\python343' + else: # pragma: win32 no cover path = '~/.pyenv/versions/3.4.3/bin/python' - expected_path = home + '/.pyenv/versions/3.4.3/bin/python' + expected_path = f'{home}/.pyenv/versions/3.4.3/bin/python' result = python.norm_version(path) assert result == expected_path -@pytest.mark.parametrize('v', ('python3.6', 'python3', 'python')) +def test_norm_version_of_default_is_sys_executable(): + assert python.norm_version('default') is None + + +@pytest.mark.parametrize('v', ('python3.9', 'python3', 'python')) def test_sys_executable_matches(v): - with mock.patch.object(sys, 'version_info', (3, 6, 7)): + with mock.patch.object(sys, 'version_info', (3, 9, 10)): assert python._sys_executable_matches(v) + assert python.norm_version(v) is None @pytest.mark.parametrize('v', ('notpython', 'python3.x')) def test_sys_executable_matches_does_not_match(v): - with mock.patch.object(sys, 'version_info', (3, 6, 7)): + with mock.patch.object(sys, 'version_info', (3, 9, 10)): assert not python._sys_executable_matches(v) + + +@pytest.mark.parametrize( + ('exe', 'realpath', 'expected'), ( + ('/usr/bin/python3', '/usr/bin/python3.7', 'python3'), + ('/usr/bin/python', '/usr/bin/python3.7', 'python3.7'), + ('/usr/bin/python', '/usr/bin/python', None), + ('/usr/bin/python3.7m', '/usr/bin/python3.7m', 'python3.7m'), + ('v/bin/python', 'v/bin/pypy', 'pypy'), + ), +) +def test_find_by_sys_executable(exe, realpath, expected): + with mock.patch.object(sys, 'executable', exe): + with mock.patch.object(os.path, 'realpath', return_value=realpath): + with mock.patch.object(python, 'find_executable', lambda x: x): + assert python._find_by_sys_executable() == expected + + +@pytest.fixture +def python_dir(tmpdir): + with tmpdir.as_cwd(): + prefix = tmpdir.join('prefix').ensure_dir() + prefix.join('setup.py').write('import setuptools; setuptools.setup()') + prefix = Prefix(str(prefix)) + yield prefix, tmpdir + + +def test_healthy_default_creator(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + # should be healthy right after creation + assert python.health_check(prefix, C.DEFAULT) is None + + # even if a `types.py` file exists, should still be healthy + tmpdir.join('types.py').ensure() + assert python.health_check(prefix, C.DEFAULT) is None + + +def test_healthy_venv_creator(python_dir): + # venv creator produces slightly different pyvenv.cfg + prefix, tmpdir = python_dir + + with envcontext((('VIRTUALENV_CREATOR', 'venv'),)): + python.install_environment(prefix, C.DEFAULT, ()) + + assert python.health_check(prefix, C.DEFAULT) is None + + +def test_unhealthy_python_goes_missing(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + exe_name = win_exe('python') + py_exe = prefix.path(python.bin_dir('py_env-default'), exe_name) + os.remove(py_exe) + + ret = python.health_check(prefix, C.DEFAULT) + assert ret == ( + f'virtualenv python version did not match created version:\n' + f'- actual version: <>\n' + f'- expected version: {python._version_info(sys.executable)}\n' + ) + + +def test_unhealthy_with_version_change(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + with open(prefix.path('py_env-default/pyvenv.cfg'), 'a+') as f: + f.write('version_info = 1.2.3\n') + + ret = python.health_check(prefix, C.DEFAULT) + assert ret == ( + f'virtualenv python version did not match created version:\n' + f'- actual version: {python._version_info(sys.executable)}\n' + f'- expected version: 1.2.3\n' + ) + + +def test_unhealthy_system_version_changes(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + with open(prefix.path('py_env-default/pyvenv.cfg'), 'a') as f: + f.write('base-executable = /does/not/exist\n') + + ret = python.health_check(prefix, C.DEFAULT) + assert ret == ( + f'base executable python version does not match created version:\n' + f'- base-executable version: <>\n' # noqa: E501 + f'- expected version: {python._version_info(sys.executable)}\n' + ) + + +def test_unhealthy_old_virtualenv(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + # simulate "old" virtualenv by deleting this file + os.remove(prefix.path('py_env-default/pyvenv.cfg')) + + ret = python.health_check(prefix, C.DEFAULT) + assert ret == 'pyvenv.cfg does not exist (old virtualenv?)' + + +def test_unhealthy_unexpected_pyvenv(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + # simulate a buggy environment build (I don't think this is possible) + with open(prefix.path('py_env-default/pyvenv.cfg'), 'w'): + pass + + ret = python.health_check(prefix, C.DEFAULT) + assert ret == "created virtualenv's pyvenv.cfg is missing `version_info`" + + +def test_unhealthy_then_replaced(python_dir): + prefix, tmpdir = python_dir + + python.install_environment(prefix, C.DEFAULT, ()) + + # simulate an exe which returns an old version + exe_name = win_exe('python') + py_exe = prefix.path(python.bin_dir('py_env-default'), exe_name) + os.rename(py_exe, f'{py_exe}.tmp') + + with open(py_exe, 'w') as f: + f.write('#!/usr/bin/env bash\necho 1.2.3\n') + make_executable(py_exe) + + # should be unhealthy due to version mismatch + ret = python.health_check(prefix, C.DEFAULT) + assert ret == ( + f'virtualenv python version did not match created version:\n' + f'- actual version: 1.2.3\n' + f'- expected version: {python._version_info(sys.executable)}\n' + ) + + # now put the exe back and it should be healthy again + os.replace(f'{py_exe}.tmp', py_exe) + + assert python.health_check(prefix, C.DEFAULT) is None + + +def test_language_versioned_python_hook(tmp_path): + setup_py = '''\ +from setuptools import setup +setup( + name='example', + py_modules=['mod'], + entry_points={'console_scripts': ['myexe=mod:main']}, +) +''' + tmp_path.joinpath('setup.py').write_text(setup_py) + tmp_path.joinpath('mod.py').write_text('def main(): print("ohai")') + + # we patch this to force virtualenv executing with `-p` since we can't + # reliably have multiple pythons available in CI + with mock.patch.object( + python, + '_sys_executable_matches', + return_value=False, + ): + assert run_language(tmp_path, python, 'myexe') == (0, b'ohai\n') + + +def _make_hello_hello(tmp_path): + setup_py = '''\ +from setuptools import setup + +setup( + name='socks', + version='0.0.0', + py_modules=['socks'], + entry_points={'console_scripts': ['socks = socks:main']}, +) +''' + + main_py = '''\ +import sys + +def main(): + print(repr(sys.argv[1:])) + print('hello hello') + return 0 +''' + tmp_path.joinpath('setup.py').write_text(setup_py) + tmp_path.joinpath('socks.py').write_text(main_py) + + +def test_simple_python_hook(tmp_path): + _make_hello_hello(tmp_path) + + ret = run_language(tmp_path, python, 'socks', [os.devnull]) + assert ret == (0, f'[{os.devnull!r}]\nhello hello\n'.encode()) + + +def test_simple_python_hook_default_version(tmp_path): + # make sure that this continues to work for platforms where default + # language detection does not work + with mock.patch.object( + python, + 'get_default_version', + return_value=C.DEFAULT, + ): + test_simple_python_hook(tmp_path) + + +def test_python_hook_weird_setup_cfg(tmp_path): + _make_hello_hello(tmp_path) + setup_cfg = '[install]\ninstall_scripts=/usr/sbin' + tmp_path.joinpath('setup.cfg').write_text(setup_cfg) + + ret = run_language(tmp_path, python, 'socks', [os.devnull]) + assert ret == (0, f'[{os.devnull!r}]\nhello hello\n'.encode()) + + +def test_local_repo_with_other_artifacts(tmp_path): + cmd_output_b('git', 'init', tmp_path) + _make_local_repo(str(tmp_path)) + # pretend a rust install also ran here + tmp_path.joinpath('target').mkdir() + + ret, out = run_language(tmp_path, python, 'python --version') + + assert ret == 0 + assert out.startswith(b'Python ') diff --git a/tests/languages/r_test.py b/tests/languages/r_test.py new file mode 100644 index 00000000..9e73129e --- /dev/null +++ b/tests/languages/r_test.py @@ -0,0 +1,297 @@ +from __future__ import annotations + +import os.path +from unittest import mock + +import pytest + +import pre_commit.constants as C +from pre_commit import envcontext +from pre_commit import lang_base +from pre_commit.languages import r +from pre_commit.prefix import Prefix +from pre_commit.store import _make_local_repo +from pre_commit.util import resource_text +from pre_commit.util import win_exe +from testing.language_helpers import run_language + + +def test_r_parsing_file_no_opts_no_args(tmp_path): + cmd = r._cmd_from_hook( + Prefix(str(tmp_path)), + 'Rscript some-script.R', + (), + is_local=False, + ) + assert cmd == ( + 'Rscript', + '--no-save', '--no-restore', '--no-site-file', '--no-environ', + str(tmp_path.joinpath('some-script.R')), + ) + + +def test_r_parsing_file_opts_no_args(): + with pytest.raises(ValueError) as excinfo: + r._entry_validate(['Rscript', '--no-init', '/path/to/file']) + + msg, = excinfo.value.args + assert msg == ( + 'The only valid syntax is `Rscript -e {expr}`' + 'or `Rscript path/to/hook/script`' + ) + + +def test_r_parsing_file_no_opts_args(tmp_path): + cmd = r._cmd_from_hook( + Prefix(str(tmp_path)), + 'Rscript some-script.R', + ('--no-cache',), + is_local=False, + ) + assert cmd == ( + 'Rscript', + '--no-save', '--no-restore', '--no-site-file', '--no-environ', + str(tmp_path.joinpath('some-script.R')), + '--no-cache', + ) + + +def test_r_parsing_expr_no_opts_no_args1(tmp_path): + cmd = r._cmd_from_hook( + Prefix(str(tmp_path)), + "Rscript -e '1+1'", + (), + is_local=False, + ) + assert cmd == ( + 'Rscript', + '--no-save', '--no-restore', '--no-site-file', '--no-environ', + '-e', '1+1', + ) + + +def test_r_parsing_local_hook_path_is_not_expanded(tmp_path): + cmd = r._cmd_from_hook( + Prefix(str(tmp_path)), + 'Rscript path/to/thing.R', + (), + is_local=True, + ) + assert cmd == ( + 'Rscript', + '--no-save', '--no-restore', '--no-site-file', '--no-environ', + 'path/to/thing.R', + ) + + +def test_r_parsing_expr_no_opts_no_args2(): + with pytest.raises(ValueError) as excinfo: + r._entry_validate(['Rscript', '-e', '1+1', '-e', 'letters']) + msg, = excinfo.value.args + assert msg == 'You can supply at most one expression.' + + +def test_r_parsing_expr_opts_no_args2(): + with pytest.raises(ValueError) as excinfo: + r._entry_validate( + ['Rscript', '--vanilla', '-e', '1+1', '-e', 'letters'], + ) + msg, = excinfo.value.args + assert msg == ( + 'The only valid syntax is `Rscript -e {expr}`' + 'or `Rscript path/to/hook/script`' + ) + + +def test_r_parsing_expr_args_in_entry2(): + with pytest.raises(ValueError) as excinfo: + r._entry_validate(['Rscript', '-e', 'expr1', '--another-arg']) + + msg, = excinfo.value.args + assert msg == 'You can supply at most one expression.' + + +def test_r_parsing_expr_non_Rscirpt(): + with pytest.raises(ValueError) as excinfo: + r._entry_validate(['AnotherScript', '-e', '{{}}']) + + msg, = excinfo.value.args + assert msg == 'entry must start with `Rscript`.' + + +def test_rscript_exec_relative_to_r_home(): + expected = os.path.join('r_home_dir', 'bin', win_exe('Rscript')) + with envcontext.envcontext((('R_HOME', 'r_home_dir'),)): + assert r._rscript_exec() == expected + + +def test_path_rscript_exec_no_r_home_set(): + with envcontext.envcontext((('R_HOME', envcontext.UNSET),)): + assert r._rscript_exec() == 'Rscript' + + +@pytest.fixture +def renv_lock_file(tmp_path): + renv_lock = '''\ +{ + "R": { + "Version": "4.0.3", + "Repositories": [ + { + "Name": "CRAN", + "URL": "https://cloud.r-project.org" + } + ] + }, + "Packages": { + "renv": { + "Package": "renv", + "Version": "0.12.5", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "5c0cdb37f063c58cdab3c7e9fbb8bd2c" + }, + "rprojroot": { + "Package": "rprojroot", + "Version": "1.0", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "86704667fe0860e4fec35afdfec137f3" + } + } +} +''' + tmp_path.joinpath('renv.lock').write_text(renv_lock) + yield + + +@pytest.fixture +def description_file(tmp_path): + description = '''\ +Package: gli.clu +Title: What the Package Does (One Line, Title Case) +Type: Package +Version: 0.0.0.9000 +Authors@R: + person(given = "First", + family = "Last", + role = c("aut", "cre"), + email = "first.last@example.com", + comment = c(ORCID = "YOUR-ORCID-ID")) +Description: What the package does (one paragraph). +License: `use_mit_license()`, `use_gpl3_license()` or friends to + pick a license +Encoding: UTF-8 +LazyData: true +Roxygen: list(markdown = TRUE) +RoxygenNote: 7.1.1 +Imports: + rprojroot +''' + tmp_path.joinpath('DESCRIPTION').write_text(description) + yield + + +@pytest.fixture +def hello_world_file(tmp_path): + hello_world = '''\ +stopifnot( + packageVersion('rprojroot') == '1.0', + packageVersion('gli.clu') == '0.0.0.9000' +) +cat("Hello, World, from R!\n") +''' + tmp_path.joinpath('hello-world.R').write_text(hello_world) + yield + + +@pytest.fixture +def renv_folder(tmp_path): + renv_dir = tmp_path.joinpath('renv') + renv_dir.mkdir() + activate_r = resource_text('empty_template_activate.R') + renv_dir.joinpath('activate.R').write_text(activate_r) + yield + + +def test_r_hook( + tmp_path, + renv_lock_file, + description_file, + hello_world_file, + renv_folder, +): + expected = (0, b'Hello, World, from R!\n') + assert run_language(tmp_path, r, 'Rscript hello-world.R') == expected + + +def test_r_inline(tmp_path): + _make_local_repo(str(tmp_path)) + + cmd = '''\ +Rscript -e ' + stopifnot(packageVersion("rprojroot") == "1.0") + cat(commandArgs(trailingOnly = TRUE), "from R!\n", sep=", ") +' +''' + + ret = run_language( + tmp_path, + r, + cmd, + deps=('rprojroot@1.0',), + args=('hi', 'hello'), + ) + assert ret == (0, b'hi, hello, from R!\n') + + +@pytest.fixture +def prefix(tmpdir): + yield Prefix(str(tmpdir)) + + +@pytest.fixture +def installed_environment( + renv_lock_file, + hello_world_file, + renv_folder, + prefix, +): + env_dir = lang_base.environment_dir( + prefix, r.ENVIRONMENT_DIR, r.get_default_version(), + ) + r.install_environment(prefix, C.DEFAULT, ()) + yield prefix, env_dir + + +def test_health_check_healthy(installed_environment): + # should be healthy right after creation + prefix, _ = installed_environment + assert r.health_check(prefix, C.DEFAULT) is None + + +def test_health_check_after_downgrade(installed_environment): + prefix, _ = installed_environment + + # pretend the saved installed version is old + with mock.patch.object(r, '_read_installed_version', return_value='1.0.0'): + output = r.health_check(prefix, C.DEFAULT) + + assert output is not None + assert output.startswith('Hooks were installed for R version') + + +@pytest.mark.parametrize('version', ('NULL', 'NA', "''")) +def test_health_check_without_version(prefix, installed_environment, version): + prefix, env_dir = installed_environment + + # simulate old pre-commit install by unsetting the installed version + r._execute_r_in_renv( + f'renv::settings$r.version({version})', + prefix=prefix, version=C.DEFAULT, cwd=env_dir, + ) + + # no R version specified fails as unhealty + msg = 'Hooks were installed with an unknown R version' + check_output = r.health_check(prefix, C.DEFAULT) + assert check_output is not None and check_output.startswith(msg) diff --git a/tests/languages/ruby_test.py b/tests/languages/ruby_test.py index a0b4cfd4..5d767b25 100644 --- a/tests/languages/ruby_test.py +++ b/tests/languages/ruby_test.py @@ -1,42 +1,139 @@ -from __future__ import unicode_literals +from __future__ import annotations -import os.path -import pipes +import tarfile +from unittest import mock -from pre_commit.languages.ruby import _install_rbenv -from pre_commit.prefix import Prefix -from pre_commit.util import cmd_output -from testing.util import xfailif_windows_no_ruby +import pytest + +import pre_commit.constants as C +from pre_commit import parse_shebang +from pre_commit.envcontext import envcontext +from pre_commit.languages import ruby +from pre_commit.languages.ruby import _resource_bytesio +from pre_commit.store import _make_local_repo +from testing.language_helpers import run_language +from testing.util import cwd +from testing.util import xfailif_windows -@xfailif_windows_no_ruby -def test_install_rbenv(tempdir_factory): - prefix = Prefix(tempdir_factory.get()) - _install_rbenv(prefix) - # Should have created rbenv directory - assert os.path.exists(prefix.path('rbenv-default')) - # We should have created our `activate` script - activate_path = prefix.path('rbenv-default', 'bin', 'activate') - assert os.path.exists(activate_path) +ACTUAL_GET_DEFAULT_VERSION = ruby.get_default_version.__wrapped__ - # Should be able to activate using our script and access rbenv - cmd_output( - 'bash', '-c', - '. {} && rbenv --help'.format( - pipes.quote(prefix.path('rbenv-default', 'bin', 'activate')), - ), + +@pytest.fixture +def find_exe_mck(): + with mock.patch.object(parse_shebang, 'find_executable') as mck: + yield mck + + +def test_uses_default_version_when_not_available(find_exe_mck): + find_exe_mck.return_value = None + assert ACTUAL_GET_DEFAULT_VERSION() == C.DEFAULT + + +def test_uses_system_if_both_gem_and_ruby_are_available(find_exe_mck): + find_exe_mck.return_value = '/path/to/exe' + assert ACTUAL_GET_DEFAULT_VERSION() == 'system' + + +@pytest.mark.parametrize( + 'filename', + ('rbenv.tar.gz', 'ruby-build.tar.gz', 'ruby-download.tar.gz'), +) +def test_archive_root_stat(filename): + with _resource_bytesio(filename) as f: + with tarfile.open(fileobj=f) as tarf: + root, _, _ = filename.partition('.') + assert oct(tarf.getmember(root).mode) == '0o755' + + +def _setup_hello_world(tmp_path): + bin_dir = tmp_path.joinpath('bin') + bin_dir.mkdir() + bin_dir.joinpath('ruby_hook').write_text( + '#!/usr/bin/env ruby\n' + "puts 'Hello world from a ruby hook'\n", + ) + gemspec = '''\ +Gem::Specification.new do |s| + s.name = 'ruby_hook' + s.version = '0.1.0' + s.authors = ['Anthony Sottile'] + s.summary = 'A ruby hook!' + s.description = 'A ruby hook!' + s.files = ['bin/ruby_hook'] + s.executables = ['ruby_hook'] +end +''' + tmp_path.joinpath('ruby_hook.gemspec').write_text(gemspec) + + +def test_ruby_hook_system(tmp_path): + assert ruby.get_default_version() == 'system' + + _setup_hello_world(tmp_path) + + ret = run_language(tmp_path, ruby, 'ruby_hook') + assert ret == (0, b'Hello world from a ruby hook\n') + + +def test_ruby_with_user_install_set(tmp_path): + gemrc = tmp_path.joinpath('gemrc') + gemrc.write_text('gem: --user-install\n') + + with envcontext((('GEMRC', str(gemrc)),)): + test_ruby_hook_system(tmp_path) + + +def test_ruby_additional_deps(tmp_path): + _make_local_repo(tmp_path) + + ret = run_language( + tmp_path, + ruby, + 'ruby -e', + args=('require "jmespath"',), + deps=('jmespath',), + ) + assert ret == (0, b'') + + +@xfailif_windows # pragma: win32 no cover +def test_ruby_hook_default(tmp_path): + _setup_hello_world(tmp_path) + + out, ret = run_language(tmp_path, ruby, 'rbenv --help', version='default') + assert out == 0 + assert ret.startswith(b'Usage: rbenv ') + + +@xfailif_windows # pragma: win32 no cover +def test_ruby_hook_language_version(tmp_path): + _setup_hello_world(tmp_path) + tmp_path.joinpath('bin', 'ruby_hook').write_text( + '#!/usr/bin/env ruby\n' + 'puts RUBY_VERSION\n' + "puts 'Hello world from a ruby hook'\n", ) + ret = run_language(tmp_path, ruby, 'ruby_hook', version='3.2.0') + assert ret == (0, b'3.2.0\nHello world from a ruby hook\n') -@xfailif_windows_no_ruby -def test_install_rbenv_with_version(tempdir_factory): - prefix = Prefix(tempdir_factory.get()) - _install_rbenv(prefix, version='1.9.3p547') - # Should be able to activate and use rbenv install - cmd_output( - 'bash', '-c', - '. {} && rbenv install --help'.format( - pipes.quote(prefix.path('rbenv-1.9.3p547', 'bin', 'activate')), - ), +@xfailif_windows # pragma: win32 no cover +def test_ruby_with_bundle_disable_shared_gems(tmp_path): + workdir = tmp_path.joinpath('workdir') + workdir.mkdir() + # this needs a `source` or there's a deprecation warning + # silencing this with `BUNDLE_GEMFILE` breaks some tools (#2739) + workdir.joinpath('Gemfile').write_text('source ""\ngem "lol_hai"\n') + # this bundle config causes things to be written elsewhere + bundle = workdir.joinpath('.bundle') + bundle.mkdir() + bundle.joinpath('config').write_text( + 'BUNDLE_DISABLE_SHARED_GEMS: true\n' + 'BUNDLE_PATH: vendor/gem\n', ) + + with cwd(workdir): + # `3.2.0` has new enough `gem` reading `.bundle` + test_ruby_hook_language_version(tmp_path) diff --git a/tests/languages/rust_test.py b/tests/languages/rust_test.py new file mode 100644 index 00000000..52e35613 --- /dev/null +++ b/tests/languages/rust_test.py @@ -0,0 +1,115 @@ +from __future__ import annotations + +from unittest import mock + +import pytest + +import pre_commit.constants as C +from pre_commit import parse_shebang +from pre_commit.languages import rust +from pre_commit.store import _make_local_repo +from testing.language_helpers import run_language +from testing.util import cwd + +ACTUAL_GET_DEFAULT_VERSION = rust.get_default_version.__wrapped__ + + +@pytest.fixture +def cmd_output_b_mck(): + with mock.patch.object(rust, 'cmd_output_b') as mck: + yield mck + + +def test_sets_system_when_rust_is_available(cmd_output_b_mck): + cmd_output_b_mck.return_value = (0, b'', b'') + assert ACTUAL_GET_DEFAULT_VERSION() == 'system' + + +def test_uses_default_when_rust_is_not_available(cmd_output_b_mck): + cmd_output_b_mck.return_value = (127, b'', b'error: not found') + assert ACTUAL_GET_DEFAULT_VERSION() == C.DEFAULT + + +def test_selects_system_even_if_rust_toolchain_toml(tmp_path): + toolchain_toml = '[toolchain]\nchannel = "wtf"\n' + tmp_path.joinpath('rust-toolchain.toml').write_text(toolchain_toml) + + with cwd(tmp_path): + assert ACTUAL_GET_DEFAULT_VERSION() == 'system' + + +def _make_hello_world(tmp_path): + src_dir = tmp_path.joinpath('src') + src_dir.mkdir() + src_dir.joinpath('main.rs').write_text( + 'fn main() {\n' + ' println!("Hello, world!");\n' + '}\n', + ) + tmp_path.joinpath('Cargo.toml').write_text( + '[package]\n' + 'name = "hello_world"\n' + 'version = "0.1.0"\n' + 'edition = "2021"\n', + ) + + +def test_installs_rust_missing_rustup(tmp_path): + _make_hello_world(tmp_path) + + # pretend like `rustup` doesn't exist so it gets bootstrapped + calls = [] + orig = parse_shebang.find_executable + + def mck(exe, env=None): + calls.append(exe) + if len(calls) == 1: + assert exe == 'rustup' + return None + return orig(exe, env=env) + + with mock.patch.object(parse_shebang, 'find_executable', side_effect=mck): + ret = run_language(tmp_path, rust, 'hello_world', version='1.56.0') + assert calls == ['rustup', 'rustup', 'cargo', 'hello_world'] + assert ret == (0, b'Hello, world!\n') + + +@pytest.mark.parametrize('version', (C.DEFAULT, '1.56.0')) +def test_language_version_with_rustup(tmp_path, version): + assert parse_shebang.find_executable('rustup') is not None + + _make_hello_world(tmp_path) + + ret = run_language(tmp_path, rust, 'hello_world', version=version) + assert ret == (0, b'Hello, world!\n') + + +@pytest.mark.parametrize('dep', ('cli:shellharden:4.2.0', 'cli:shellharden')) +def test_rust_cli_additional_dependencies(tmp_path, dep): + _make_local_repo(str(tmp_path)) + + t_sh = tmp_path.joinpath('t.sh') + t_sh.write_text('echo $hi\n') + + assert rust.get_default_version() == 'system' + ret = run_language( + tmp_path, + rust, + 'shellharden --transform', + deps=(dep,), + args=(str(t_sh),), + ) + assert ret == (0, b'echo "$hi"\n') + + +def test_run_lib_additional_dependencies(tmp_path): + _make_hello_world(tmp_path) + + deps = ('shellharden:4.2.0', 'git-version') + ret = run_language(tmp_path, rust, 'hello_world', deps=deps) + assert ret == (0, b'Hello, world!\n') + + bin_dir = tmp_path.joinpath('rustenv-system', 'bin') + assert bin_dir.is_dir() + assert not bin_dir.joinpath('shellharden').exists() + assert not bin_dir.joinpath('shellharden.exe').exists() diff --git a/tests/languages/swift_test.py b/tests/languages/swift_test.py new file mode 100644 index 00000000..e0a8ea42 --- /dev/null +++ b/tests/languages/swift_test.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import sys + +import pytest + +from pre_commit.languages import swift +from testing.language_helpers import run_language + + +@pytest.mark.skipif( + sys.platform == 'win32', + reason='swift is not supported on windows', +) +def test_swift_language(tmp_path): # pragma: win32 no cover + package_swift = '''\ +// swift-tools-version:5.0 +import PackageDescription + +let package = Package( + name: "swift_hooks_repo", + targets: [.target(name: "swift_hooks_repo")] +) +''' + tmp_path.joinpath('Package.swift').write_text(package_swift) + src_dir = tmp_path.joinpath('Sources/swift_hooks_repo') + src_dir.mkdir(parents=True) + src_dir.joinpath('main.swift').write_text('print("Hello, world!")\n') + + expected = (0, b'Hello, world!\n') + assert run_language(tmp_path, swift, 'swift_hooks_repo') == expected diff --git a/tests/languages/unsupported_script_test.py b/tests/languages/unsupported_script_test.py new file mode 100644 index 00000000..b15b67e7 --- /dev/null +++ b/tests/languages/unsupported_script_test.py @@ -0,0 +1,14 @@ +from __future__ import annotations + +from pre_commit.languages import unsupported_script +from pre_commit.util import make_executable +from testing.language_helpers import run_language + + +def test_unsupported_script_language(tmp_path): + exe = tmp_path.joinpath('main') + exe.write_text('#!/usr/bin/env bash\necho hello hello world\n') + make_executable(exe) + + expected = (0, b'hello hello world\n') + assert run_language(tmp_path, unsupported_script, 'main') == expected diff --git a/tests/languages/unsupported_test.py b/tests/languages/unsupported_test.py new file mode 100644 index 00000000..7f8461e0 --- /dev/null +++ b/tests/languages/unsupported_test.py @@ -0,0 +1,10 @@ +from __future__ import annotations + +from pre_commit.languages import unsupported +from testing.language_helpers import run_language + + +def test_unsupported_language(tmp_path): + expected = (0, b'hello hello world\n') + ret = run_language(tmp_path, unsupported, 'echo hello hello world') + assert ret == expected diff --git a/tests/logging_handler_test.py b/tests/logging_handler_test.py index 0e72541a..dc43a99f 100644 --- a/tests/logging_handler_test.py +++ b/tests/logging_handler_test.py @@ -1,27 +1,23 @@ -from __future__ import unicode_literals +from __future__ import annotations + +import logging from pre_commit import color from pre_commit.logging_handler import LoggingHandler -class FakeLogRecord(object): - def __init__(self, message, levelname, levelno): - self.message = message - self.levelname = levelname - self.levelno = levelno - - def getMessage(self): - return self.message +def _log_record(message, level): + return logging.LogRecord('name', level, '', 1, message, {}, None) def test_logging_handler_color(cap_out): handler = LoggingHandler(True) - handler.emit(FakeLogRecord('hi', 'WARNING', 30)) + handler.emit(_log_record('hi', logging.WARNING)) ret = cap_out.get() - assert ret == color.YELLOW + '[WARNING]' + color.NORMAL + ' hi\n' + assert ret == f'{color.YELLOW}[WARNING]{color.NORMAL} hi\n' def test_logging_handler_no_color(cap_out): handler = LoggingHandler(False) - handler.emit(FakeLogRecord('hi', 'WARNING', 30)) + handler.emit(_log_record('hi', logging.WARNING)) assert cap_out.get() == '[WARNING] hi\n' diff --git a/tests/main_test.py b/tests/main_test.py index 75fd5600..fed085fc 100644 --- a/tests/main_test.py +++ b/tests/main_test.py @@ -1,32 +1,36 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import argparse +import contextlib import os.path +from unittest import mock -import mock import pytest import pre_commit.constants as C from pre_commit import main -from pre_commit.error_handler import FatalError +from pre_commit.commands import hazmat +from pre_commit.errors import FatalError +from pre_commit.util import cmd_output from testing.auto_namedtuple import auto_namedtuple +from testing.util import cwd -class Args(object): - def __init__(self, **kwargs): - kwargs.setdefault('command', 'help') - kwargs.setdefault('config', C.CONFIG_FILE) - self.__dict__.update(kwargs) +def _args(**kwargs): + kwargs.setdefault('command', 'help') + kwargs.setdefault('config', C.CONFIG_FILE) + if kwargs['command'] in {'run', 'try-repo'}: + kwargs.setdefault('commit_msg_filename', None) + return argparse.Namespace(**kwargs) def test_adjust_args_and_chdir_not_in_git_dir(in_tmpdir): with pytest.raises(FatalError): - main._adjust_args_and_chdir(Args()) + main._adjust_args_and_chdir(_args()) def test_adjust_args_and_chdir_noop(in_git_dir): - args = Args(command='run', files=['f1', 'f2']) + args = _args(command='run', files=['f1', 'f2']) main._adjust_args_and_chdir(args) assert os.getcwd() == in_git_dir assert args.config == C.CONFIG_FILE @@ -35,49 +39,69 @@ def test_adjust_args_and_chdir_noop(in_git_dir): def test_adjust_args_and_chdir_relative_things(in_git_dir): in_git_dir.join('foo/cfg.yaml').ensure() - in_git_dir.join('foo').chdir() + with in_git_dir.join('foo').as_cwd(): + args = _args(command='run', files=['f1', 'f2'], config='cfg.yaml') + main._adjust_args_and_chdir(args) + assert os.getcwd() == in_git_dir + assert args.config == os.path.join('foo', 'cfg.yaml') + assert args.files == [ + os.path.join('foo', 'f1'), + os.path.join('foo', 'f2'), + ] - args = Args(command='run', files=['f1', 'f2'], config='cfg.yaml') - main._adjust_args_and_chdir(args) - assert os.getcwd() == in_git_dir - assert args.config == os.path.join('foo', 'cfg.yaml') - assert args.files == [os.path.join('foo', 'f1'), os.path.join('foo', 'f2')] + +def test_adjust_args_and_chdir_relative_commit_msg(in_git_dir): + in_git_dir.join('foo/cfg.yaml').ensure() + with in_git_dir.join('foo').as_cwd(): + args = _args(command='run', files=[], commit_msg_filename='t.txt') + main._adjust_args_and_chdir(args) + assert os.getcwd() == in_git_dir + assert args.commit_msg_filename == os.path.join('foo', 't.txt') + + +@pytest.mark.skipif(os.name != 'nt', reason='windows feature') +def test_install_on_subst(in_git_dir, store): # pragma: posix no cover + assert not os.path.exists('Z:') + cmd_output('subst', 'Z:', str(in_git_dir)) + try: + with cwd('Z:'): + test_adjust_args_and_chdir_noop('Z:\\') + finally: + cmd_output('subst', '/d', 'Z:') def test_adjust_args_and_chdir_non_relative_config(in_git_dir): - in_git_dir.join('foo').ensure_dir().chdir() - - args = Args() - main._adjust_args_and_chdir(args) - assert os.getcwd() == in_git_dir - assert args.config == C.CONFIG_FILE + with in_git_dir.join('foo').ensure_dir().as_cwd(): + args = _args() + main._adjust_args_and_chdir(args) + assert os.getcwd() == in_git_dir + assert args.config == C.CONFIG_FILE def test_adjust_args_try_repo_repo_relative(in_git_dir): - in_git_dir.join('foo').ensure_dir().chdir() - - args = Args(command='try-repo', repo='../foo', files=[]) - assert os.path.exists(args.repo) - main._adjust_args_and_chdir(args) - assert os.getcwd() == in_git_dir - assert os.path.exists(args.repo) - assert args.repo == 'foo' + with in_git_dir.join('foo').ensure_dir().as_cwd(): + args = _args(command='try-repo', repo='../foo', files=[]) + assert args.repo is not None + assert os.path.exists(args.repo) + main._adjust_args_and_chdir(args) + assert os.getcwd() == in_git_dir + assert os.path.exists(args.repo) + assert args.repo == 'foo' FNS = ( - 'autoupdate', 'clean', 'gc', 'install', 'install_hooks', 'migrate_config', - 'run', 'sample_config', 'uninstall', + 'autoupdate', 'clean', 'gc', 'hook_impl', 'install', 'install_hooks', + 'migrate_config', 'run', 'sample_config', 'uninstall', + 'validate_config', 'validate_manifest', ) CMDS = tuple(fn.replace('_', '-') for fn in FNS) @pytest.fixture def mock_commands(): - mcks = {fn: mock.patch.object(main, fn).start() for fn in FNS} - ret = auto_namedtuple(**mcks) - yield ret - for mck in ret: - mck.stop() + with contextlib.ExitStack() as ctx: + mcks = {f: ctx.enter_context(mock.patch.object(main, f)) for f in FNS} + yield auto_namedtuple(**mcks) @pytest.fixture @@ -134,6 +158,17 @@ def test_all_cmds(command, mock_commands, mock_store_dir): assert_only_one_mock_called(mock_commands) +def test_hazmat(mock_store_dir): + with mock.patch.object(hazmat, 'impl') as mck: + main.main(('hazmat', 'cd', 'subdir', '--', 'cmd', '--', 'f1', 'f2')) + assert mck.call_count == 1 + (arg,), dct = mck.call_args + assert dct == {} + assert arg.tool == 'cd' + assert arg.subdir == 'subdir' + assert arg.cmd == ['cmd', '--', 'f1', 'f2'] + + def test_try_repo(mock_store_dir): with mock.patch.object(main, 'try_repo') as patch: main.main(('try-repo', '.')) @@ -143,7 +178,28 @@ def test_try_repo(mock_store_dir): def test_init_templatedir(mock_store_dir): with mock.patch.object(main, 'init_templatedir') as patch: main.main(('init-templatedir', 'tdir')) + assert patch.call_count == 1 + assert 'tdir' in patch.call_args[0] + assert patch.call_args[1]['hook_types'] is None + assert patch.call_args[1]['skip_on_missing_config'] is True + + +def test_init_templatedir_options(mock_store_dir): + args = ( + 'init-templatedir', + 'tdir', + '--hook-type', + 'commit-msg', + '--no-allow-missing-config', + ) + with mock.patch.object(main, 'init_templatedir') as patch: + main.main(args) + + assert patch.call_count == 1 + assert 'tdir' in patch.call_args[0] + assert patch.call_args[1]['hook_types'] == ['commit-msg'] + assert patch.call_args[1]['skip_on_missing_config'] is False def test_help_cmd_in_empty_directory( @@ -164,13 +220,16 @@ def test_expected_fatal_error_no_git_repo(in_tmpdir, cap_out, mock_store_dir): with pytest.raises(SystemExit): main.main([]) log_file = os.path.join(mock_store_dir, 'pre-commit.log') - assert cap_out.get() == ( + cap_out_lines = cap_out.get().splitlines() + assert ( + cap_out_lines[-2] == 'An error has occurred: FatalError: git failed. ' - 'Is it installed, and are you in a Git repository directory?\n' - 'Check the log at {}\n'.format(log_file) + 'Is it installed, and are you in a Git repository directory?' ) + assert cap_out_lines[-1] == f'Check the log at {log_file}' -def test_warning_on_tags_only(mock_commands, cap_out, mock_store_dir): - main.main(('autoupdate', '--tags-only')) - assert '--tags-only is the default' in cap_out.get() +def test_hook_stage_migration(mock_store_dir): + with mock.patch.object(main, 'run') as mck: + main.main(('run', '--hook-stage', 'commit')) + assert mck.call_args[0][2].hook_stage == 'pre-commit' diff --git a/tests/make_archives_test.py b/tests/make_archives_test.py deleted file mode 100644 index 52c9c9b6..00000000 --- a/tests/make_archives_test.py +++ /dev/null @@ -1,49 +0,0 @@ -from __future__ import absolute_import -from __future__ import unicode_literals - -import tarfile - -from pre_commit import git -from pre_commit import make_archives -from pre_commit.util import cmd_output -from testing.util import git_commit - - -def test_make_archive(in_git_dir, tmpdir): - output_dir = tmpdir.join('output').ensure_dir() - # Add a files to the git directory - in_git_dir.join('foo').ensure() - cmd_output('git', 'add', '.') - git_commit() - # We'll use this rev - head_rev = git.head_rev('.') - # And check that this file doesn't exist - in_git_dir.join('bar').ensure() - cmd_output('git', 'add', '.') - git_commit() - - # Do the thing - archive_path = make_archives.make_archive( - 'foo', in_git_dir.strpath, head_rev, output_dir.strpath, - ) - - expected = output_dir.join('foo.tar.gz') - assert archive_path == expected.strpath - assert expected.exists() - - extract_dir = tmpdir.join('extract').ensure_dir() - with tarfile.open(archive_path) as tf: - tf.extractall(extract_dir.strpath) - - # Verify the contents of the tar - assert extract_dir.join('foo').isdir() - assert extract_dir.join('foo/foo').exists() - assert not extract_dir.join('foo/.git').exists() - assert not extract_dir.join('foo/bar').exists() - - -def test_main(tmpdir): - make_archives.main(('--dest', tmpdir.strpath)) - - for archive, _, _ in make_archives.REPOS: - assert tmpdir.join('{}.tar.gz'.format(archive)).exists() diff --git a/tests/meta_hooks/check_hooks_apply_test.py b/tests/meta_hooks/check_hooks_apply_test.py index 06bdd045..63f97152 100644 --- a/tests/meta_hooks/check_hooks_apply_test.py +++ b/tests/meta_hooks/check_hooks_apply_test.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from pre_commit.meta_hooks import check_hooks_apply from testing.fixtures import add_config_to_repo diff --git a/tests/meta_hooks/check_useless_excludes_test.py b/tests/meta_hooks/check_useless_excludes_test.py index d261e814..15b68b4c 100644 --- a/tests/meta_hooks/check_useless_excludes_test.py +++ b/tests/meta_hooks/check_useless_excludes_test.py @@ -1,5 +1,12 @@ +from __future__ import annotations + +from pre_commit import git from pre_commit.meta_hooks import check_useless_excludes +from pre_commit.util import cmd_output from testing.fixtures import add_config_to_repo +from testing.fixtures import make_config_from_repo +from testing.fixtures import make_repo +from testing.util import xfailif_windows def test_useless_exclude_global(capsys, in_git_dir): @@ -113,3 +120,20 @@ def test_valid_exclude(capsys, in_git_dir): out, _ = capsys.readouterr() assert out == '' + + +@xfailif_windows # pragma: win32 no cover +def test_useless_excludes_broken_symlink(capsys, in_git_dir, tempdir_factory): + path = make_repo(tempdir_factory, 'script_hooks_repo') + config = make_config_from_repo(path) + config['hooks'][0]['exclude'] = 'broken-symlink' + add_config_to_repo(in_git_dir.strpath, config) + + in_git_dir.join('broken-symlink').mksymlinkto('DNE') + cmd_output('git', 'add', 'broken-symlink') + git.commit() + + assert check_useless_excludes.main(('.pre-commit-config.yaml',)) == 0 + + out, _ = capsys.readouterr() + assert out == '' diff --git a/tests/meta_hooks/identity_test.py b/tests/meta_hooks/identity_test.py index 3eff00be..97c20ea6 100644 --- a/tests/meta_hooks/identity_test.py +++ b/tests/meta_hooks/identity_test.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from pre_commit.meta_hooks import identity diff --git a/tests/output_test.py b/tests/output_test.py index 8b6ea90d..c806829a 100644 --- a/tests/output_test.py +++ b/tests/output_test.py @@ -1,86 +1,11 @@ -from __future__ import unicode_literals +from __future__ import annotations -import mock -import pytest +import io -from pre_commit import color from pre_commit import output -@pytest.mark.parametrize( - 'kwargs', - ( - # both end_msg and end_len - {'end_msg': 'end', 'end_len': 1, 'end_color': '', 'use_color': True}, - # Neither end_msg nor end_len - {}, - # Neither color option for end_msg - {'end_msg': 'end'}, - # No use_color for end_msg - {'end_msg': 'end', 'end_color': ''}, - # No end_color for end_msg - {'end_msg': 'end', 'use_color': ''}, - ), -) -def test_get_hook_message_raises(kwargs): - with pytest.raises(ValueError): - output.get_hook_message('start', **kwargs) - - -def test_case_with_end_len(): - ret = output.get_hook_message('start', end_len=5, cols=15) - assert ret == 'start' + '.' * 4 - - -def test_case_with_end_msg(): - ret = output.get_hook_message( - 'start', - end_msg='end', - end_color='', - use_color=False, - cols=15, - ) - assert ret == 'start' + '.' * 6 + 'end' + '\n' - - -def test_case_with_end_msg_using_color(): - ret = output.get_hook_message( - 'start', - end_msg='end', - end_color=color.RED, - use_color=True, - cols=15, - ) - assert ret == 'start' + '.' * 6 + color.RED + 'end' + color.NORMAL + '\n' - - -def test_case_with_postfix_message(): - ret = output.get_hook_message( - 'start', - postfix='post ', - end_msg='end', - end_color='', - use_color=False, - cols=20, - ) - assert ret == 'start' + '.' * 6 + 'post ' + 'end' + '\n' - - -def test_make_sure_postfix_is_not_colored(): - ret = output.get_hook_message( - 'start', - postfix='post ', - end_msg='end', - end_color=color.RED, - use_color=True, - cols=20, - ) - assert ret == ( - 'start' + '.' * 6 + 'post ' + color.RED + 'end' + color.NORMAL + '\n' - ) - - def test_output_write_writes(): - fake_stream = mock.Mock() - output.write('hello world', fake_stream) - assert fake_stream.write.call_count == 1 + stream = io.BytesIO() + output.write('hello world', stream) + assert stream.getvalue() == b'hello world' diff --git a/tests/parse_shebang_test.py b/tests/parse_shebang_test.py index 400a287c..bd4384df 100644 --- a/tests/parse_shebang_test.py +++ b/tests/parse_shebang_test.py @@ -1,10 +1,8 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import contextlib -import distutils.spawn -import io -import os +import os.path +import shutil import sys import pytest @@ -15,15 +13,21 @@ from pre_commit.envcontext import Var from pre_commit.util import make_executable +def _echo_exe() -> str: + exe = shutil.which('echo') + assert exe is not None + return exe + + def test_file_doesnt_exist(): assert parse_shebang.parse_filename('herp derp derp') == () def test_simple_case(tmpdir): x = tmpdir.join('f') - x.write_text('#!/usr/bin/env python', encoding='UTF-8') + x.write('#!/usr/bin/env echo') make_executable(x.strpath) - assert parse_shebang.parse_filename(x.strpath) == ('python',) + assert parse_shebang.parse_filename(x.strpath) == ('echo',) def test_find_executable_full_path(): @@ -31,8 +35,7 @@ def test_find_executable_full_path(): def test_find_executable_on_path(): - expected = distutils.spawn.find_executable('echo') - assert parse_shebang.find_executable('echo') == expected + assert parse_shebang.find_executable('echo') == _echo_exe() def test_find_executable_not_found_none(): @@ -42,8 +45,8 @@ def test_find_executable_not_found_none(): def write_executable(shebang, filename='run'): os.mkdir('bin') path = os.path.join('bin', filename) - with io.open(path, 'w') as f: - f.write('#!{}'.format(shebang)) + with open(path, 'w') as f: + f.write(f'#!{shebang}') make_executable(path) return path @@ -72,10 +75,10 @@ def test_find_executable_path_ext(in_tmpdir): env_path = {'PATH': os.path.dirname(exe_path)} env_path_ext = dict(env_path, PATHEXT=os.pathsep.join(('.exe', '.myext'))) assert parse_shebang.find_executable('run') is None - assert parse_shebang.find_executable('run', _environ=env_path) is None - ret = parse_shebang.find_executable('run.myext', _environ=env_path) + assert parse_shebang.find_executable('run', env=env_path) is None + ret = parse_shebang.find_executable('run.myext', env=env_path) assert ret == exe_path - ret = parse_shebang.find_executable('run', _environ=env_path_ext) + ret = parse_shebang.find_executable('run', env=env_path_ext) assert ret == exe_path @@ -91,6 +94,14 @@ def test_normexe_does_not_exist_sep(): assert excinfo.value.args == ('Executable `./i-dont-exist-lol` not found',) +@pytest.mark.xfail(sys.platform == 'win32', reason='posix only') +def test_normexe_not_executable(tmpdir): # pragma: win32 no cover + tmpdir.join('exe').ensure() + with tmpdir.as_cwd(), pytest.raises(OSError) as excinfo: + parse_shebang.normexe('./exe') + assert excinfo.value.args == ('Executable `./exe` is not executable',) + + def test_normexe_is_a_directory(tmpdir): with tmpdir.as_cwd(): tmpdir.join('exe').ensure_dir() @@ -98,7 +109,7 @@ def test_normexe_is_a_directory(tmpdir): with pytest.raises(OSError) as excinfo: parse_shebang.normexe(exe) msg, = excinfo.value.args - assert msg == 'Executable `{}` is a directory'.format(exe) + assert msg == f'Executable `{exe}` is a directory' def test_normexe_already_full_path(): @@ -106,39 +117,38 @@ def test_normexe_already_full_path(): def test_normexe_gives_full_path(): - expected = distutils.spawn.find_executable('echo') - assert parse_shebang.normexe('echo') == expected - assert os.sep in expected + assert parse_shebang.normexe('echo') == _echo_exe() + assert os.sep in _echo_exe() def test_normalize_cmd_trivial(): - cmd = (distutils.spawn.find_executable('echo'), 'hi') + cmd = (_echo_exe(), 'hi') assert parse_shebang.normalize_cmd(cmd) == cmd def test_normalize_cmd_PATH(): - cmd = ('python', '--version') - expected = (distutils.spawn.find_executable('python'), '--version') + cmd = ('echo', '--version') + expected = (_echo_exe(), '--version') assert parse_shebang.normalize_cmd(cmd) == expected def test_normalize_cmd_shebang(in_tmpdir): - python = distutils.spawn.find_executable('python').replace(os.sep, '/') - path = write_executable(python) - assert parse_shebang.normalize_cmd((path,)) == (python, path) + us = sys.executable.replace(os.sep, '/') + path = write_executable(us) + assert parse_shebang.normalize_cmd((path,)) == (us, path) def test_normalize_cmd_PATH_shebang_full_path(in_tmpdir): - python = distutils.spawn.find_executable('python').replace(os.sep, '/') - path = write_executable(python) + us = sys.executable.replace(os.sep, '/') + path = write_executable(us) with bin_on_path(): ret = parse_shebang.normalize_cmd(('run',)) - assert ret == (python, os.path.abspath(path)) + assert ret == (us, os.path.abspath(path)) def test_normalize_cmd_PATH_shebang_PATH(in_tmpdir): - python = distutils.spawn.find_executable('python') - path = write_executable('/usr/bin/env python') + echo = _echo_exe() + path = write_executable('/usr/bin/env echo') with bin_on_path(): ret = parse_shebang.normalize_cmd(('run',)) - assert ret == (python, os.path.abspath(path)) + assert ret == (echo, os.path.abspath(path)) diff --git a/tests/prefix_test.py b/tests/prefix_test.py index 2806cff1..1eac087d 100644 --- a/tests/prefix_test.py +++ b/tests/prefix_test.py @@ -1,4 +1,4 @@ -from __future__ import unicode_literals +from __future__ import annotations import os.path diff --git a/tests/repository_test.py b/tests/repository_test.py index 03ffeb07..5d71c3e4 100644 --- a/tests/repository_test.py +++ b/tests/repository_test.py @@ -1,48 +1,49 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import os.path -import re +import shlex import shutil import sys +from typing import Any +from unittest import mock import cfgv -import mock import pytest import pre_commit.constants as C -from pre_commit import five -from pre_commit import parse_shebang +from pre_commit import lang_base +from pre_commit.all_languages import languages from pre_commit.clientlib import CONFIG_SCHEMA from pre_commit.clientlib import load_manifest -from pre_commit.envcontext import envcontext -from pre_commit.languages import golang -from pre_commit.languages import helpers -from pre_commit.languages import node -from pre_commit.languages import pcre +from pre_commit.hook import Hook from pre_commit.languages import python -from pre_commit.languages import ruby -from pre_commit.languages import rust +from pre_commit.languages import unsupported from pre_commit.prefix import Prefix +from pre_commit.repository import _hook_installed from pre_commit.repository import all_hooks -from pre_commit.repository import Hook from pre_commit.repository import install_hook_envs from pre_commit.util import cmd_output +from pre_commit.util import cmd_output_b from testing.fixtures import make_config_from_repo from testing.fixtures import make_repo -from testing.fixtures import modify_manifest +from testing.language_helpers import run_language from testing.util import cwd from testing.util import get_resource_path -from testing.util import skipif_cant_run_docker -from testing.util import skipif_cant_run_swift -from testing.util import xfailif_broken_deep_listdir -from testing.util import xfailif_no_pcre_support -from testing.util import xfailif_no_venv -from testing.util import xfailif_windows_no_ruby -def _norm_out(b): - return b.replace(b'\r\n', b'\n') +def _hook_run(hook, filenames, color): + return run_language( + path=hook.prefix.prefix_dir, + language=languages[hook.language], + exe=hook.entry, + args=hook.args, + file_args=filenames, + version=hook.language_version, + deps=hook.additional_dependencies, + is_local=hook.src == 'local', + require_serial=hook.require_serial, + color=color, + ) def _get_hook_no_install(repo_config, store, hook_id): @@ -50,7 +51,7 @@ def _get_hook_no_install(repo_config, store, hook_id): config = cfgv.validate(config, CONFIG_SCHEMA) config = cfgv.apply_defaults(config, CONFIG_SCHEMA) hooks = all_hooks(config, store) - hook, = [hook for hook in hooks if hook.id == hook_id] + hook, = (hook for hook in hooks if hook.id == hook_id) return hook @@ -69,257 +70,14 @@ def _test_hook_repo( expected, expected_return_code=0, config_kwargs=None, + color=False, ): path = make_repo(tempdir_factory, repo_path) config = make_config_from_repo(path, **(config_kwargs or {})) - ret = _get_hook(config, store, hook_id).run(args) - assert ret[0] == expected_return_code - assert _norm_out(ret[1]) == expected - - -def test_python_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'python_hooks_repo', - 'foo', [os.devnull], - b"['" + five.to_bytes(os.devnull) + b"']\nHello World\n", - ) - - -def test_python_hook_default_version(tempdir_factory, store): - # make sure that this continues to work for platforms where default - # language detection does not work - with mock.patch.object( - python, 'get_default_version', return_value=C.DEFAULT, - ): - test_python_hook(tempdir_factory, store) - - -def test_python_hook_args_with_spaces(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'python_hooks_repo', - 'foo', - [], - b"['i have spaces', 'and\"\\'quotes', '$and !this']\n" - b'Hello World\n', - config_kwargs={ - 'hooks': [{ - 'id': 'foo', - 'args': ['i have spaces', 'and"\'quotes', '$and !this'], - }], - }, - ) - - -def test_python_hook_weird_setup_cfg(in_git_dir, tempdir_factory, store): - in_git_dir.join('setup.cfg').write('[install]\ninstall_scripts=/usr/sbin') - - _test_hook_repo( - tempdir_factory, store, 'python_hooks_repo', - 'foo', [os.devnull], - b"['" + five.to_bytes(os.devnull) + b"']\nHello World\n", - ) - - -@xfailif_no_venv -def test_python_venv(tempdir_factory, store): # pragma: no cover (no venv) - _test_hook_repo( - tempdir_factory, store, 'python_venv_hooks_repo', - 'foo', [os.devnull], - b"['" + five.to_bytes(os.devnull) + b"']\nHello World\n", - ) - - -def test_switch_language_versions_doesnt_clobber(tempdir_factory, store): - # We're using the python3 repo because it prints the python version - path = make_repo(tempdir_factory, 'python3_hooks_repo') - - def run_on_version(version, expected_output): - config = make_config_from_repo(path) - config['hooks'][0]['language_version'] = version - ret = _get_hook(config, store, 'python3-hook').run([]) - assert ret[0] == 0 - assert _norm_out(ret[1]) == expected_output - - run_on_version('python2', b'2\n[]\nHello World\n') - run_on_version('python3', b'3\n[]\nHello World\n') - - -def test_versioned_python_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'python3_hooks_repo', - 'python3-hook', - [os.devnull], - b"3\n['" + five.to_bytes(os.devnull) + b"']\nHello World\n", - ) - - -@skipif_cant_run_docker # pragma: windows no cover -def test_run_a_docker_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'docker_hooks_repo', - 'docker-hook', - ['Hello World from docker'], b'Hello World from docker\n', - ) - - -@skipif_cant_run_docker # pragma: windows no cover -def test_run_a_docker_hook_with_entry_args(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'docker_hooks_repo', - 'docker-hook-arg', - ['Hello World from docker'], b'Hello World from docker', - ) - - -@skipif_cant_run_docker # pragma: windows no cover -def test_run_a_failing_docker_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'docker_hooks_repo', - 'docker-hook-failing', - ['Hello World from docker'], b'', - expected_return_code=1, - ) - - -@skipif_cant_run_docker # pragma: windows no cover -@pytest.mark.parametrize('hook_id', ('echo-entrypoint', 'echo-cmd')) -def test_run_a_docker_image_hook(tempdir_factory, store, hook_id): - _test_hook_repo( - tempdir_factory, store, 'docker_image_hooks_repo', - hook_id, - ['Hello World from docker'], b'Hello World from docker\n', - ) - - -@xfailif_broken_deep_listdir -def test_run_a_node_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'node_hooks_repo', - 'foo', [os.devnull], b'Hello World\n', - ) - - -@xfailif_broken_deep_listdir -def test_run_versioned_node_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'node_versioned_hooks_repo', - 'versioned-node-hook', [os.devnull], b'v9.3.0\nHello World\n', - ) - - -@xfailif_windows_no_ruby -def test_run_a_ruby_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'ruby_hooks_repo', - 'ruby_hook', [os.devnull], b'Hello world from a ruby hook\n', - ) - - -@xfailif_windows_no_ruby -def test_run_versioned_ruby_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'ruby_versioned_hooks_repo', - 'ruby_hook', - [os.devnull], - b'2.5.1\nHello world from a ruby hook\n', - ) - - -@xfailif_windows_no_ruby -def test_run_ruby_hook_with_disable_shared_gems( - tempdir_factory, - store, - tmpdir, -): - """Make sure a Gemfile in the project doesn't interfere.""" - tmpdir.join('Gemfile').write('gem "lol_hai"') - tmpdir.join('.bundle').mkdir() - tmpdir.join('.bundle', 'config').write( - 'BUNDLE_DISABLE_SHARED_GEMS: true\n' - 'BUNDLE_PATH: vendor/gem\n', - ) - with cwd(tmpdir.strpath): - _test_hook_repo( - tempdir_factory, store, 'ruby_versioned_hooks_repo', - 'ruby_hook', - [os.devnull], - b'2.5.1\nHello world from a ruby hook\n', - ) - - -def test_system_hook_with_spaces(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'system_hook_with_spaces_repo', - 'system-hook-with-spaces', [os.devnull], b'Hello World\n', - ) - - -@skipif_cant_run_swift # pragma: windows no cover -def test_swift_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'swift_hooks_repo', - 'swift-hooks-repo', [], b'Hello, world!\n', - ) - - -def test_golang_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'golang_hooks_repo', - 'golang-hook', [], b'hello world\n', - ) - - -def test_golang_hook_still_works_when_gobin_is_set(tempdir_factory, store): - gobin_dir = tempdir_factory.get() - with envcontext([('GOBIN', gobin_dir)]): - test_golang_hook(tempdir_factory, store) - assert os.listdir(gobin_dir) == [] - - -def test_rust_hook(tempdir_factory, store): - _test_hook_repo( - tempdir_factory, store, 'rust_hooks_repo', - 'rust-hook', [], b'hello world\n', - ) - - -@pytest.mark.parametrize('dep', ('cli:shellharden:3.1.0', 'cli:shellharden')) -def test_additional_rust_cli_dependencies_installed( - tempdir_factory, store, dep, -): - path = make_repo(tempdir_factory, 'rust_hooks_repo') - config = make_config_from_repo(path) - # A small rust package with no dependencies. - config['hooks'][0]['additional_dependencies'] = [dep] - hook = _get_hook(config, store, 'rust-hook') - binaries = os.listdir( - hook.prefix.path( - helpers.environment_dir(rust.ENVIRONMENT_DIR, C.DEFAULT), 'bin', - ), - ) - # normalize for windows - binaries = [os.path.splitext(binary)[0] for binary in binaries] - assert 'shellharden' in binaries - - -def test_additional_rust_lib_dependencies_installed( - tempdir_factory, store, -): - path = make_repo(tempdir_factory, 'rust_hooks_repo') - config = make_config_from_repo(path) - # A small rust package with no dependencies. - deps = ['shellharden:3.1.0'] - config['hooks'][0]['additional_dependencies'] = deps - hook = _get_hook(config, store, 'rust-hook') - binaries = os.listdir( - hook.prefix.path( - helpers.environment_dir(rust.ENVIRONMENT_DIR, C.DEFAULT), 'bin', - ), - ) - # normalize for windows - binaries = [os.path.splitext(binary)[0] for binary in binaries] - assert 'rust-hello-world' in binaries - assert 'shellharden' not in binaries + hook = _get_hook(config, store, hook_id) + ret, out = _hook_run(hook, args, color=color) + assert ret == expected_return_code + assert out == expected def test_missing_executable(tempdir_factory, store): @@ -362,86 +120,31 @@ def test_run_hook_with_curly_braced_arguments(tempdir_factory, store): ) -def _make_grep_repo(language, entry, store, args=()): - config = { - 'repo': 'local', - 'hooks': [{ - 'id': 'grep-hook', - 'name': 'grep-hook', - 'language': language, - 'entry': entry, - 'args': args, - 'types': ['text'], - }], - } - return _get_hook(config, store, 'grep-hook') +def test_intermixed_stdout_stderr(tempdir_factory, store): + _test_hook_repo( + tempdir_factory, store, 'stdout_stderr_repo', + 'stdout-stderr', + [], + b'0\n1\n2\n3\n4\n5\n', + ) -@pytest.fixture -def greppable_files(tmpdir): - with tmpdir.as_cwd(): - cmd_output('git', 'init', '.') - tmpdir.join('f1').write_binary(b"hello'hi\nworld\n") - tmpdir.join('f2').write_binary(b'foo\nbar\nbaz\n') - tmpdir.join('f3').write_binary(b'[WARN] hi\n') - yield tmpdir - - -class TestPygrep(object): - language = 'pygrep' - - def test_grep_hook_matching(self, greppable_files, store): - hook = _make_grep_repo(self.language, 'ello', store) - ret, out, _ = hook.run(('f1', 'f2', 'f3')) - assert ret == 1 - assert _norm_out(out) == b"f1:1:hello'hi\n" - - def test_grep_hook_case_insensitive(self, greppable_files, store): - hook = _make_grep_repo(self.language, 'ELLO', store, args=['-i']) - ret, out, _ = hook.run(('f1', 'f2', 'f3')) - assert ret == 1 - assert _norm_out(out) == b"f1:1:hello'hi\n" - - @pytest.mark.parametrize('regex', ('nope', "foo'bar", r'^\[INFO\]')) - def test_grep_hook_not_matching(self, regex, greppable_files, store): - hook = _make_grep_repo(self.language, regex, store) - ret, out, _ = hook.run(('f1', 'f2', 'f3')) - assert (ret, out) == (0, b'') - - -@xfailif_no_pcre_support # pragma: windows no cover -class TestPCRE(TestPygrep): - """organized as a class for xfailing pcre""" - language = 'pcre' - - def test_pcre_hook_many_files(self, greppable_files, store): - # This is intended to simulate lots of passing files and one failing - # file to make sure it still fails. This is not the case when naively - # using a system hook with `grep -H -n '...'` - hook = _make_grep_repo('pcre', 'ello', store) - ret, out, _ = hook.run((os.devnull,) * 15000 + ('f1',)) - assert ret == 1 - assert _norm_out(out) == b"f1:1:hello'hi\n" - - def test_missing_pcre_support(self, greppable_files, store): - def no_grep(exe, **kwargs): - assert exe == pcre.GREP - return None - - with mock.patch.object(parse_shebang, 'find_executable', no_grep): - hook = _make_grep_repo('pcre', 'ello', store) - ret, out, _ = hook.run(('f1', 'f2', 'f3')) - assert ret == 1 - expected = 'Executable `{}` not found'.format(pcre.GREP).encode() - assert out == expected +@pytest.mark.xfail(sys.platform == 'win32', reason='ptys are posix-only') +def test_output_isatty(tempdir_factory, store): + _test_hook_repo( + tempdir_factory, store, 'stdout_stderr_repo', + 'tty-check', + [], + b'stdin: False\nstdout: True\nstderr: True\n', + color=True, + ) def _norm_pwd(path): # Under windows bash's temp and windows temp is different. # This normalizes to the bash /tmp - return cmd_output( - 'bash', '-c', "cd '{}' && pwd".format(path), - encoding=None, + return cmd_output_b( + 'bash', '-c', f"cd '{path}' && pwd", )[1].strip() @@ -480,103 +183,22 @@ def test_additional_dependencies_roll_forward(tempdir_factory, store): assert 'mccabe' not in cmd_output('pip', 'freeze', '-l')[1] -@xfailif_windows_no_ruby # pragma: windows no cover -def test_additional_ruby_dependencies_installed(tempdir_factory, store): - path = make_repo(tempdir_factory, 'ruby_hooks_repo') - config = make_config_from_repo(path) - config['hooks'][0]['additional_dependencies'] = ['tins'] - hook = _get_hook(config, store, 'ruby_hook') - with ruby.in_env(hook.prefix, hook.language_version): - output = cmd_output('gem', 'list', '--local')[1] - assert 'tins' in output +@pytest.mark.parametrize('v', ('v1', 'v2')) +def test_repository_state_compatibility(tempdir_factory, store, v): + path = make_repo(tempdir_factory, 'python_hooks_repo') - -@xfailif_broken_deep_listdir # pragma: windows no cover -def test_additional_node_dependencies_installed(tempdir_factory, store): - path = make_repo(tempdir_factory, 'node_hooks_repo') config = make_config_from_repo(path) - # Careful to choose a small package that's not depped by npm - config['hooks'][0]['additional_dependencies'] = ['lodash'] hook = _get_hook(config, store, 'foo') - with node.in_env(hook.prefix, hook.language_version): - output = cmd_output('npm', 'ls', '-g')[1] - assert 'lodash' in output - - -def test_additional_golang_dependencies_installed( - tempdir_factory, store, -): - path = make_repo(tempdir_factory, 'golang_hooks_repo') - config = make_config_from_repo(path) - # A small go package - deps = ['github.com/golang/example/hello'] - config['hooks'][0]['additional_dependencies'] = deps - hook = _get_hook(config, store, 'golang-hook') - binaries = os.listdir( - hook.prefix.path( - helpers.environment_dir(golang.ENVIRONMENT_DIR, C.DEFAULT), 'bin', - ), + envdir = lang_base.environment_dir( + hook.prefix, + python.ENVIRONMENT_DIR, + hook.language_version, ) - # normalize for windows - binaries = [os.path.splitext(binary)[0] for binary in binaries] - assert 'hello' in binaries + os.remove(os.path.join(envdir, f'.install_state_{v}')) + assert _hook_installed(hook) is True -def test_local_golang_additional_dependencies(store): - config = { - 'repo': 'local', - 'hooks': [{ - 'id': 'hello', - 'name': 'hello', - 'entry': 'hello', - 'language': 'golang', - 'additional_dependencies': ['github.com/golang/example/hello'], - }], - } - ret = _get_hook(config, store, 'hello').run(()) - assert ret[0] == 0 - assert _norm_out(ret[1]) == b'Hello, Go examples!\n' - - -def test_local_rust_additional_dependencies(store): - config = { - 'repo': 'local', - 'hooks': [{ - 'id': 'hello', - 'name': 'hello', - 'entry': 'hello', - 'language': 'rust', - 'additional_dependencies': ['cli:hello-cli:0.2.2'], - }], - } - ret = _get_hook(config, store, 'hello').run(()) - assert ret[0] == 0 - assert _norm_out(ret[1]) == b'Hello World!\n' - - -def test_fail_hooks(store): - config = { - 'repo': 'local', - 'hooks': [{ - 'id': 'fail', - 'name': 'fail', - 'language': 'fail', - 'entry': 'make sure to name changelogs as .rst!', - 'files': r'changelog/.*(? too-much: foo, hello' - assert fake_log_handler.handle.call_args[0][0].msg == expected + msg, = caplog.messages + assert msg == 'Unexpected key(s) present on local => too-much: foo, hello' -def test_reinstall(tempdir_factory, store, log_info_mock): +def test_reinstall(tempdir_factory, store, caplog): path = make_repo(tempdir_factory, 'python_hooks_repo') config = make_config_from_repo(path) _get_hook(config, store, 'foo') # We print some logging during clone (1) + install (3) - assert log_info_mock.call_count == 4 - log_info_mock.reset_mock() + assert len(caplog.record_tuples) == 4 + caplog.clear() # Reinstall on another run should not trigger another install _get_hook(config, store, 'foo') - assert log_info_mock.call_count == 0 + assert len(caplog.record_tuples) == 0 def test_control_c_control_c_on_install(tempdir_factory, store): @@ -619,7 +241,7 @@ def test_control_c_control_c_on_install(tempdir_factory, store): # raise as well. with pytest.raises(MyKeyboardInterrupt): with mock.patch.object( - helpers, 'run_setup_cmd', side_effect=MyKeyboardInterrupt, + lang_base, 'setup_cmd', side_effect=MyKeyboardInterrupt, ): with mock.patch.object( shutil, 'rmtree', side_effect=MyKeyboardInterrupt, @@ -628,15 +250,19 @@ def test_control_c_control_c_on_install(tempdir_factory, store): # Should have made an environment, however this environment is broken! hook, = hooks - assert hook.prefix.exists( - helpers.environment_dir(python.ENVIRONMENT_DIR, hook.language_version), + envdir = lang_base.environment_dir( + hook.prefix, + python.ENVIRONMENT_DIR, + hook.language_version, ) + assert os.path.exists(envdir) + # However, it should be perfectly runnable (reinstall after botched # install) install_hook_envs(hooks, store) - retv, stdout, stderr = hook.run(()) - assert retv == 0 + ret, out = _hook_run(hook, (), color=False) + assert ret == 0 def test_invalidated_virtualenv(tempdir_factory, store): @@ -647,24 +273,27 @@ def test_invalidated_virtualenv(tempdir_factory, store): hook = _get_hook(config, store, 'foo') # Simulate breaking of the virtualenv - libdir = hook.prefix.path( - helpers.environment_dir(python.ENVIRONMENT_DIR, hook.language_version), - 'lib', hook.language_version, + envdir = lang_base.environment_dir( + hook.prefix, + python.ENVIRONMENT_DIR, + hook.language_version, ) + libdir = os.path.join(envdir, 'lib', hook.language_version) paths = [ os.path.join(libdir, p) for p in ('site.py', 'site.pyc', '__pycache__') ] - cmd_output('rm', '-rf', *paths) + cmd_output_b('rm', '-rf', *paths) # pre-commit should rebuild the virtualenv and it should be runnable - retv, stdout, stderr = _get_hook(config, store, 'foo').run(()) - assert retv == 0 + hook = _get_hook(config, store, 'foo') + ret, out = _hook_run(hook, (), color=False) + assert ret == 0 def test_really_long_file_paths(tempdir_factory, store): base_path = tempdir_factory.get() really_long_path = os.path.join(base_path, 'really_long' * 10) - cmd_output('git', 'init', really_long_path) + cmd_output_b('git', 'init', really_long_path) path = make_repo(tempdir_factory, 'python_hooks_repo') config = make_config_from_repo(path) @@ -687,7 +316,7 @@ def test_config_overrides_repo_specifics(tempdir_factory, store): def _create_repo_with_tags(tempdir_factory, src, tag): path = make_repo(tempdir_factory, src) - cmd_output('git', 'tag', tag, cwd=path) + cmd_output_b('git', 'tag', tag, cwd=path) return path @@ -697,14 +326,16 @@ def test_tags_on_repositories(in_tmpdir, tempdir_factory, store): git2 = _create_repo_with_tags(tempdir_factory, 'script_hooks_repo', tag) config1 = make_config_from_repo(git1, rev=tag) - ret1 = _get_hook(config1, store, 'prints_cwd').run(('-L',)) - assert ret1[0] == 0 - assert ret1[1].strip() == _norm_pwd(in_tmpdir) + hook1 = _get_hook(config1, store, 'prints_cwd') + ret1, out1 = _hook_run(hook1, ('-L',), color=False) + assert ret1 == 0 + assert out1.strip() == _norm_pwd(in_tmpdir) config2 = make_config_from_repo(git2, rev=tag) - ret2 = _get_hook(config2, store, 'bash_hook').run(('bar',)) - assert ret2[0] == 0 - assert ret2[1] == b'bar\nHello World\n' + hook2 = _get_hook(config2, store, 'bash_hook') + ret2, out2 = _hook_run(hook2, ('bar',), color=False) + assert ret2 == 0 + assert out2 == b'bar\nHello World\n' @pytest.fixture @@ -718,23 +349,19 @@ def local_python_config(): return {'repo': 'local', 'hooks': hooks} -@pytest.mark.xfail( # pragma: windows no cover - sys.platform == 'win32', - reason='microsoft/azure-pipelines-image-generation#989', -) def test_local_python_repo(store, local_python_config): hook = _get_hook(local_python_config, store, 'foo') # language_version should have been adjusted to the interpreter version assert hook.language_version != C.DEFAULT - ret = hook.run(('filename',)) - assert ret[0] == 0 - assert _norm_out(ret[1]) == b"['filename']\nHello World\n" + ret, out = _hook_run(hook, ('filename',), color=False) + assert ret == 0 + assert out == b"['filename']\nHello World\n" def test_default_language_version(store, local_python_config): - config = { + config: dict[str, Any] = { 'default_language_version': {'python': 'fake'}, - 'default_stages': ['commit'], + 'default_stages': ['pre-commit'], 'repos': [local_python_config], } @@ -749,68 +376,43 @@ def test_default_language_version(store, local_python_config): def test_default_stages(store, local_python_config): - config = { + config: dict[str, Any] = { 'default_language_version': {'python': C.DEFAULT}, - 'default_stages': ['commit'], + 'default_stages': ['pre-commit'], 'repos': [local_python_config], } # `stages` was not set, should default hook, = all_hooks(config, store) - assert hook.stages == ['commit'] + assert hook.stages == ['pre-commit'] # `stages` is set, should not default - config['repos'][0]['hooks'][0]['stages'] = ['push'] + config['repos'][0]['hooks'][0]['stages'] = ['pre-push'] hook, = all_hooks(config, store) - assert hook.stages == ['push'] + assert hook.stages == ['pre-push'] -def test_hook_id_not_present(tempdir_factory, store, fake_log_handler): +def test_hook_id_not_present(tempdir_factory, store, caplog): path = make_repo(tempdir_factory, 'script_hooks_repo') config = make_config_from_repo(path) config['hooks'][0]['id'] = 'i-dont-exist' with pytest.raises(SystemExit): _get_hook(config, store, 'i-dont-exist') - assert fake_log_handler.handle.call_args[0][0].msg == ( - '`i-dont-exist` is not present in repository file://{}. ' - 'Typo? Perhaps it is introduced in a newer version? ' - 'Often `pre-commit autoupdate` fixes this.'.format(path) + _, msg = caplog.messages + assert msg == ( + f'`i-dont-exist` is not present in repository file://{path}. ' + f'Typo? Perhaps it is introduced in a newer version? ' + f'Often `pre-commit autoupdate` fixes this.' ) -def test_too_new_version(tempdir_factory, store, fake_log_handler): - path = make_repo(tempdir_factory, 'script_hooks_repo') - with modify_manifest(path) as manifest: - manifest[0]['minimum_pre_commit_version'] = '999.0.0' - config = make_config_from_repo(path) - with pytest.raises(SystemExit): - _get_hook(config, store, 'bash_hook') - msg = fake_log_handler.handle.call_args[0][0].msg - assert re.match( - r'^The hook `bash_hook` requires pre-commit version 999\.0\.0 but ' - r'version \d+\.\d+\.\d+ is installed. ' - r'Perhaps run `pip install --upgrade pre-commit`\.$', - msg, - ) - - -@pytest.mark.parametrize('version', ('0.1.0', C.VERSION)) -def test_versions_ok(tempdir_factory, store, version): - path = make_repo(tempdir_factory, 'script_hooks_repo') - with modify_manifest(path) as manifest: - manifest[0]['minimum_pre_commit_version'] = version - config = make_config_from_repo(path) - # Should succeed - _get_hook(config, store, 'bash_hook') - - def test_manifest_hooks(tempdir_factory, store): path = make_repo(tempdir_factory, 'script_hooks_repo') config = make_config_from_repo(path) hook = _get_hook(config, store, 'bash_hook') assert hook == Hook( - src='file://{}'.format(path), + src=f'file://{path}', prefix=Prefix(mock.ANY), additional_dependencies=[], alias='', @@ -822,16 +424,96 @@ def test_manifest_hooks(tempdir_factory, store): exclude_types=[], files='', id='bash_hook', - language='script', + language='unsupported_script', language_version='default', log_file='', minimum_pre_commit_version='0', name='Bash hook', pass_filenames=True, require_serial=False, - stages=( - 'commit', 'prepare-commit-msg', 'commit-msg', 'manual', 'push', - ), + stages=[ + 'commit-msg', + 'post-checkout', + 'post-commit', + 'post-merge', + 'post-rewrite', + 'pre-commit', + 'pre-merge-commit', + 'pre-push', + 'pre-rebase', + 'prepare-commit-msg', + 'manual', + ], types=['file'], + types_or=[], verbose=False, + fail_fast=False, ) + + +def test_non_installable_hook_error_for_language_version(store, caplog): + config = { + 'repo': 'local', + 'hooks': [{ + 'id': 'system-hook', + 'name': 'system-hook', + 'language': 'unsupported', + 'entry': 'python3 -c "import sys; print(sys.version)"', + 'language_version': 'python3.10', + }], + } + with pytest.raises(SystemExit) as excinfo: + _get_hook(config, store, 'system-hook') + assert excinfo.value.code == 1 + + msg, = caplog.messages + assert msg == ( + 'The hook `system-hook` specifies `language_version` but is using ' + 'language `unsupported` which does not install an environment. ' + 'Perhaps you meant to use a specific language?' + ) + + +def test_non_installable_hook_error_for_additional_dependencies(store, caplog): + config = { + 'repo': 'local', + 'hooks': [{ + 'id': 'system-hook', + 'name': 'system-hook', + 'language': 'unsupported', + 'entry': 'python3 -c "import sys; print(sys.version)"', + 'additional_dependencies': ['astpretty'], + }], + } + with pytest.raises(SystemExit) as excinfo: + _get_hook(config, store, 'system-hook') + assert excinfo.value.code == 1 + + msg, = caplog.messages + assert msg == ( + 'The hook `system-hook` specifies `additional_dependencies` but is ' + 'using language `unsupported` which does not install an environment. ' + 'Perhaps you meant to use a specific language?' + ) + + +def test_args_with_spaces_and_quotes(tmp_path): + ret = run_language( + tmp_path, unsupported, + f"{shlex.quote(sys.executable)} -c 'import sys; print(sys.argv[1:])'", + ('i have spaces', 'and"\'quotes', '$and !this'), + ) + + expected = b"['i have spaces', 'and\"\\'quotes', '$and !this']\n" + assert ret == (0, expected) + + +def test_hazmat(tmp_path): + ret = run_language( + tmp_path, unsupported, + f'pre-commit hazmat ignore-exit-code {shlex.quote(sys.executable)} ' + f"-c 'import sys; raise SystemExit(sys.argv[1:])'", + ('f1', 'f2'), + ) + expected = b"['f1', 'f2']\n" + assert ret == (0, expected) diff --git a/tests/staged_files_only_test.py b/tests/staged_files_only_test.py index 2410bffe..cd2f6387 100644 --- a/tests/staged_files_only_test.py +++ b/tests/staged_files_only_test.py @@ -1,15 +1,15 @@ -# -*- coding: UTF-8 -*- -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import io +import contextlib import itertools import os.path import shutil import pytest +import re_assert from pre_commit import git +from pre_commit.errors import FatalError from pre_commit.staged_files_only import staged_files_only from pre_commit.util import cmd_output from testing.auto_namedtuple import auto_namedtuple @@ -17,6 +17,7 @@ from testing.fixtures import git_dir from testing.util import cwd from testing.util import get_resource_path from testing.util import git_commit +from testing.util import xfailif_windows FOO_CONTENTS = '\n'.join(('1', '2', '3', '4', '5', '6', '7', '8', '')) @@ -29,7 +30,8 @@ def patch_dir(tempdir_factory): def get_short_git_status(): git_status = cmd_output('git', 'status', '-s')[1] - return dict(reversed(line.split()) for line in git_status.splitlines()) + line_parts = [line.split() for line in git_status.splitlines()] + return {v: k for k, v in line_parts} @pytest.fixture @@ -47,7 +49,7 @@ def _test_foo_state( encoding='UTF-8', ): assert os.path.exists(path.foo_filename) - with io.open(path.foo_filename, encoding=encoding) as f: + with open(path.foo_filename, encoding=encoding) as f: assert f.read() == foo_contents actual_status = get_short_git_status()['foo'] assert status == actual_status @@ -64,7 +66,7 @@ def test_foo_nothing_unstaged(foo_staged, patch_dir): def test_foo_something_unstaged(foo_staged, patch_dir): - with io.open(foo_staged.foo_filename, 'w') as foo_file: + with open(foo_staged.foo_filename, 'w') as foo_file: foo_file.write('herp\nderp\n') _test_foo_state(foo_staged, 'herp\nderp\n', 'AM') @@ -76,7 +78,7 @@ def test_foo_something_unstaged(foo_staged, patch_dir): def test_does_not_crash_patch_dir_does_not_exist(foo_staged, patch_dir): - with io.open(foo_staged.foo_filename, 'w') as foo_file: + with open(foo_staged.foo_filename, 'w') as foo_file: foo_file.write('hello\nworld\n') shutil.rmtree(patch_dir) @@ -97,25 +99,25 @@ def test_foo_something_unstaged_diff_color_always(foo_staged, patch_dir): def test_foo_both_modify_non_conflicting(foo_staged, patch_dir): - with io.open(foo_staged.foo_filename, 'w') as foo_file: - foo_file.write(FOO_CONTENTS + '9\n') + with open(foo_staged.foo_filename, 'w') as foo_file: + foo_file.write(f'{FOO_CONTENTS}9\n') - _test_foo_state(foo_staged, FOO_CONTENTS + '9\n', 'AM') + _test_foo_state(foo_staged, f'{FOO_CONTENTS}9\n', 'AM') with staged_files_only(patch_dir): _test_foo_state(foo_staged) # Modify the file as part of the "pre-commit" - with io.open(foo_staged.foo_filename, 'w') as foo_file: + with open(foo_staged.foo_filename, 'w') as foo_file: foo_file.write(FOO_CONTENTS.replace('1', 'a')) _test_foo_state(foo_staged, FOO_CONTENTS.replace('1', 'a'), 'AM') - _test_foo_state(foo_staged, FOO_CONTENTS.replace('1', 'a') + '9\n', 'AM') + _test_foo_state(foo_staged, f'{FOO_CONTENTS.replace("1", "a")}9\n', 'AM') def test_foo_both_modify_conflicting(foo_staged, patch_dir): - with io.open(foo_staged.foo_filename, 'w') as foo_file: + with open(foo_staged.foo_filename, 'w') as foo_file: foo_file.write(FOO_CONTENTS.replace('1', 'a')) _test_foo_state(foo_staged, FOO_CONTENTS.replace('1', 'a'), 'AM') @@ -124,7 +126,7 @@ def test_foo_both_modify_conflicting(foo_staged, patch_dir): _test_foo_state(foo_staged) # Modify in the same place as the stashed diff - with io.open(foo_staged.foo_filename, 'w') as foo_file: + with open(foo_staged.foo_filename, 'w') as foo_file: foo_file.write(FOO_CONTENTS.replace('1', 'b')) _test_foo_state(foo_staged, FOO_CONTENTS.replace('1', 'b'), 'AM') @@ -142,8 +144,8 @@ def img_staged(in_git_dir): def _test_img_state(path, expected_file='img1.jpg', status='A'): assert os.path.exists(path.img_filename) - with io.open(path.img_filename, 'rb') as f1: - with io.open(get_resource_path(expected_file), 'rb') as f2: + with open(path.img_filename, 'rb') as f1: + with open(get_resource_path(expected_file), 'rb') as f2: assert f1.read() == f2.read() actual_status = get_short_git_status()['img.jpg'] assert status == actual_status @@ -185,9 +187,11 @@ def test_img_conflict(img_staged, patch_dir): @pytest.fixture -def submodule_with_commits(tempdir_factory): +def repo_with_commits(tempdir_factory): path = git_dir(tempdir_factory) with cwd(path): + open('foo', 'a+').close() + cmd_output('git', 'add', 'foo') git_commit() rev1 = cmd_output('git', 'rev-parse', 'HEAD')[1].strip() git_commit() @@ -200,18 +204,21 @@ def checkout_submodule(rev): @pytest.fixture -def sub_staged(submodule_with_commits, tempdir_factory): +def sub_staged(repo_with_commits, tempdir_factory): path = git_dir(tempdir_factory) with cwd(path): + open('bar', 'a+').close() + cmd_output('git', 'add', 'bar') + git_commit() cmd_output( - 'git', 'submodule', 'add', submodule_with_commits.path, 'sub', + 'git', 'submodule', 'add', repo_with_commits.path, 'sub', ) - checkout_submodule(submodule_with_commits.rev1) + checkout_submodule(repo_with_commits.rev1) cmd_output('git', 'add', 'sub') yield auto_namedtuple( path=path, sub_path=os.path.join(path, 'sub'), - submodule=submodule_with_commits, + submodule=repo_with_commits, ) @@ -246,9 +253,37 @@ def test_sub_something_unstaged(sub_staged, patch_dir): _test_sub_state(sub_staged, 'rev2', 'AM') +def test_submodule_does_not_discard_changes(sub_staged, patch_dir): + with open('bar', 'w') as f: + f.write('unstaged changes') + + foo_path = os.path.join(sub_staged.sub_path, 'foo') + with open(foo_path, 'w') as f: + f.write('foo contents') + + with staged_files_only(patch_dir): + with open('bar') as f: + assert f.read() == '' + + with open(foo_path) as f: + assert f.read() == 'foo contents' + + with open('bar') as f: + assert f.read() == 'unstaged changes' + + with open(foo_path) as f: + assert f.read() == 'foo contents' + + +def test_submodule_does_not_discard_changes_recurse(sub_staged, patch_dir): + cmd_output('git', 'config', 'submodule.recurse', '1', cwd=sub_staged.path) + + test_submodule_does_not_discard_changes(sub_staged, patch_dir) + + def test_stage_utf8_changes(foo_staged, patch_dir): contents = '\u2603' - with io.open('foo', 'w', encoding='UTF-8') as foo_file: + with open('foo', 'w', encoding='UTF-8') as foo_file: foo_file.write(contents) _test_foo_state(foo_staged, contents, 'AM') @@ -260,7 +295,7 @@ def test_stage_utf8_changes(foo_staged, patch_dir): def test_stage_non_utf8_changes(foo_staged, patch_dir): contents = 'ΓΊ' # Produce a latin-1 diff - with io.open('foo', 'w', encoding='latin-1') as foo_file: + with open('foo', 'w', encoding='latin-1') as foo_file: foo_file.write(contents) _test_foo_state(foo_staged, contents, 'AM', encoding='latin-1') @@ -282,14 +317,14 @@ def test_non_utf8_conflicting_diff(foo_staged, patch_dir): # Previously, the error message (though discarded immediately) was being # decoded with the UTF-8 codec (causing a crash) contents = 'ΓΊ \n' - with io.open('foo', 'w', encoding='latin-1') as foo_file: + with open('foo', 'w', encoding='latin-1') as foo_file: foo_file.write(contents) _test_foo_state(foo_staged, contents, 'AM', encoding='latin-1') with staged_files_only(patch_dir): _test_foo_state(foo_staged) # Create a conflicting diff that will need to be rolled back - with io.open('foo', 'w') as foo_file: + with open('foo', 'w') as foo_file: foo_file.write('') _test_foo_state(foo_staged, contents, 'AM', encoding='latin-1') @@ -323,12 +358,27 @@ def test_crlf(in_git_dir, patch_dir, crlf_before, crlf_after, autocrlf): assert_no_diff() +@pytest.mark.parametrize('autocrlf', ('true', 'input')) +def test_crlf_diff_only(in_git_dir, patch_dir, autocrlf): + # due to a quirk (?) in git -- a diff only in crlf does not show but + # still results in an exit code of `1` + # we treat this as "no diff" -- though ideally it would discard the diff + # while committing + cmd_output('git', 'config', '--local', 'core.autocrlf', autocrlf) + + _write(b'1\r\n2\r\n3\r\n') + cmd_output('git', 'add', 'foo') + _write(b'1\n2\n3\n') + with staged_files_only(patch_dir): + pass + + def test_whitespace_errors(in_git_dir, patch_dir): cmd_output('git', 'config', '--local', 'apply.whitespace', 'error') test_crlf(in_git_dir, patch_dir, True, True, 'true') -def test_autocrlf_commited_crlf(in_git_dir, patch_dir): +def test_autocrlf_committed_crlf(in_git_dir, patch_dir): """Regression test for #570""" cmd_output('git', 'config', '--local', 'core.autocrlf', 'false') _write(b'1\r\n2\r\n') @@ -351,3 +401,49 @@ def test_intent_to_add(in_git_dir, patch_dir): with staged_files_only(patch_dir): assert_no_diff() assert git.intent_to_add_files() == ['foo'] + + +@contextlib.contextmanager +def _unreadable(f): + orig = os.stat(f).st_mode + os.chmod(f, 0o000) + try: + yield + finally: + os.chmod(f, orig) + + +@xfailif_windows # pragma: win32 no cover +def test_failed_diff_does_not_discard_changes(in_git_dir, patch_dir): + # stage 3 files + for i in range(3): + with open(str(i), 'w') as f: + f.write(str(i)) + cmd_output('git', 'add', '0', '1', '2') + + # modify all of their contents + for i in range(3): + with open(str(i), 'w') as f: + f.write('new contents') + + with _unreadable('1'): + with pytest.raises(FatalError) as excinfo: + with staged_files_only(patch_dir): + raise AssertionError('should have errored on enter') + + # the diff command failed to produce a diff of `1` + msg, = excinfo.value.args + re_assert.Matches( + r'^pre-commit failed to diff -- perhaps due to permissions\?\n\n' + r'command: .*\n' + r'return code: 128\n' + r'stdout: \(none\)\n' + r'stderr:\n' + r' error: open\("1"\): Permission denied\n' + r' fatal: cannot hash 1$', + ).assert_matches(msg) + + # even though it errored, the unstaged changes should still be present + for i in range(3): + with open(str(i)) as f: + assert f.read() == 'new contents' diff --git a/tests/store_test.py b/tests/store_test.py index 1833dee7..13f198ea 100644 --- a/tests/store_test.py +++ b/tests/store_test.py @@ -1,22 +1,36 @@ -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations -import io +import logging import os.path +import shlex import sqlite3 +import stat +from unittest import mock -import mock import pytest -import six +import pre_commit.constants as C from pre_commit import git from pre_commit.store import _get_default_directory +from pre_commit.store import _LOCAL_RESOURCES from pre_commit.store import Store from pre_commit.util import CalledProcessError from pre_commit.util import cmd_output from testing.fixtures import git_dir from testing.util import cwd from testing.util import git_commit +from testing.util import xfailif_windows + + +def _select_all_configs(store: Store) -> list[str]: + with store.connect() as db: + rows = db.execute('SELECT * FROM configs').fetchall() + return [path for path, in rows] + + +def _select_all_repos(store: Store) -> list[tuple[str, str, str]]: + with store.connect() as db: + return db.execute('SELECT repo, ref, path FROM repos').fetchall() def test_our_session_fixture_works(): @@ -30,7 +44,8 @@ def test_our_session_fixture_works(): def test_get_default_directory_defaults_to_home(): # Not we use the module level one which is not mocked ret = _get_default_directory() - assert ret == os.path.join(os.path.expanduser('~/.cache'), 'pre-commit') + expected = os.path.realpath(os.path.expanduser('~/.cache/pre-commit')) + assert ret == expected def test_adheres_to_xdg_specification(): @@ -38,7 +53,8 @@ def test_adheres_to_xdg_specification(): os.environ, {'XDG_CACHE_HOME': '/tmp/fakehome'}, ): ret = _get_default_directory() - assert ret == os.path.join('/tmp/fakehome', 'pre-commit') + expected = os.path.realpath('/tmp/fakehome/pre-commit') + assert ret == expected def test_uses_environment_variable_when_present(): @@ -46,14 +62,15 @@ def test_uses_environment_variable_when_present(): os.environ, {'PRE_COMMIT_HOME': '/tmp/pre_commit_home'}, ): ret = _get_default_directory() - assert ret == '/tmp/pre_commit_home' + expected = os.path.realpath('/tmp/pre_commit_home') + assert ret == expected def test_store_init(store): # Should create the store directory assert os.path.exists(store.directory) # Should create a README file indicating what the directory is about - with io.open(os.path.join(store.directory, 'README')) as readme_file: + with open(os.path.join(store.directory, 'README')) as readme_file: readme_contents = readme_file.read() for text_line in ( 'This directory is maintained by the pre-commit project.', @@ -62,7 +79,7 @@ def test_store_init(store): assert text_line in readme_contents -def test_clone(store, tempdir_factory, log_info_mock): +def test_clone(store, tempdir_factory, caplog): path = git_dir(tempdir_factory) with cwd(path): git_commit() @@ -71,7 +88,7 @@ def test_clone(store, tempdir_factory, log_info_mock): ret = store.clone(path, rev) # Should have printed some stuff - assert log_info_mock.call_args_list[0][0][0].startswith( + assert caplog.record_tuples[0][-1].startswith( 'Initializing environment for ', ) @@ -85,7 +102,73 @@ def test_clone(store, tempdir_factory, log_info_mock): assert git.head_rev(ret) == rev # Assert there's an entry in the sqlite db for this - assert store.select_all_repos() == [(path, rev, ret)] + assert _select_all_repos(store) == [(path, rev, ret)] + + +def test_warning_for_deprecated_stages_on_init(store, tempdir_factory, caplog): + manifest = '''\ +- id: hook1 + name: hook1 + language: system + entry: echo hook1 + stages: [commit, push] +- id: hook2 + name: hook2 + language: system + entry: echo hook2 + stages: [push, merge-commit] +''' + + path = git_dir(tempdir_factory) + with open(os.path.join(path, C.MANIFEST_FILE), 'w') as f: + f.write(manifest) + cmd_output('git', 'add', '.', cwd=path) + git_commit(cwd=path) + rev = git.head_rev(path) + + store.clone(path, rev) + assert caplog.record_tuples[1] == ( + 'pre_commit', + logging.WARNING, + f'repo `{path}` uses deprecated stage names ' + f'(commit, push, merge-commit) which will be removed in a future ' + f'version. ' + f'Hint: often `pre-commit autoupdate --repo {shlex.quote(path)}` ' + f'will fix this. ' + f'if it does not -- consider reporting an issue to that repo.', + ) + + # should not re-warn + caplog.clear() + store.clone(path, rev) + assert caplog.record_tuples == [] + + +def test_no_warning_for_non_deprecated_stages_on_init( + store, tempdir_factory, caplog, +): + manifest = '''\ +- id: hook1 + name: hook1 + language: system + entry: echo hook1 + stages: [pre-commit, pre-push] +- id: hook2 + name: hook2 + language: system + entry: echo hook2 + stages: [pre-push, pre-merge-commit] +''' + + path = git_dir(tempdir_factory) + with open(os.path.join(path, C.MANIFEST_FILE), 'w') as f: + f.write(manifest) + cmd_output('git', 'add', '.', cwd=path) + git_commit(cwd=path) + rev = git.head_rev(path) + + store.clone(path, rev) + assert logging.WARNING not in {tup[1] for tup in caplog.record_tuples} def test_clone_cleans_up_on_checkout_failure(store): @@ -93,7 +176,7 @@ def test_clone_cleans_up_on_checkout_failure(store): # This raises an exception because you can't clone something that # doesn't exist! store.clone('/i_dont_exist_lol', 'fake_rev') - assert '/i_dont_exist_lol' in six.text_type(excinfo.value) + assert '/i_dont_exist_lol' in str(excinfo.value) repo_dirs = [ d for d in os.listdir(store.directory) if d.startswith('repo') @@ -115,7 +198,7 @@ def test_clone_when_repo_already_exists(store): def test_clone_shallow_failure_fallback_to_complete( store, tempdir_factory, - log_info_mock, + caplog, ): path = git_dir(tempdir_factory) with cwd(path): @@ -125,13 +208,13 @@ def test_clone_shallow_failure_fallback_to_complete( # Force shallow clone failure def fake_shallow_clone(self, *args, **kwargs): - raise CalledProcessError(None, None, None) + raise CalledProcessError(1, (), b'', None) store._shallow_clone = fake_shallow_clone ret = store.clone(path, rev) # Should have printed some stuff - assert log_info_mock.call_args_list[0][0][0].startswith( + assert caplog.record_tuples[0][-1].startswith( 'Initializing environment for ', ) @@ -145,7 +228,7 @@ def test_clone_shallow_failure_fallback_to_complete( assert git.head_rev(ret) == rev # Assert there's an entry in the sqlite db for this - assert store.select_all_repos() == [(path, rev, ret)] + assert _select_all_repos(store) == [(path, rev, ret)] def test_clone_tag_not_on_mainline(store, tempdir_factory): @@ -177,23 +260,23 @@ def test_create_when_store_already_exists(store): def test_db_repo_name(store): assert store.db_repo_name('repo', ()) == 'repo' - assert store.db_repo_name('repo', ('b', 'a', 'c')) == 'repo:a,b,c' + assert store.db_repo_name('repo', ('b', 'a', 'c')) == 'repo:b,a,c' def test_local_resources_reflects_reality(): on_disk = { - res[len('empty_template_'):] + res.removeprefix('empty_template_') for res in os.listdir('pre_commit/resources') if res.startswith('empty_template_') } - assert on_disk == set(Store.LOCAL_RESOURCES) + assert on_disk == {os.path.basename(x) for x in _LOCAL_RESOURCES} def test_mark_config_as_used(store, tmpdir): with tmpdir.as_cwd(): f = tmpdir.join('f').ensure() store.mark_config_used('f') - assert store.select_all_configs() == [f.strpath] + assert _select_all_configs(store) == [f.strpath] def test_mark_config_as_used_idempotent(store, tmpdir): @@ -203,19 +286,58 @@ def test_mark_config_as_used_idempotent(store, tmpdir): def test_mark_config_as_used_does_not_exist(store): store.mark_config_used('f') - assert store.select_all_configs() == [] - - -def _simulate_pre_1_14_0(store): - with store.connect() as db: - db.executescript('DROP TABLE configs') - - -def test_select_all_configs_roll_forward(store): - _simulate_pre_1_14_0(store) - assert store.select_all_configs() == [] + assert _select_all_configs(store) == [] def test_mark_config_as_used_roll_forward(store, tmpdir): - _simulate_pre_1_14_0(store) + with store.connect() as db: # simulate pre-1.14.0 + db.executescript('DROP TABLE configs') test_mark_config_as_used(store, tmpdir) + + +@xfailif_windows # pragma: win32 no cover +def test_mark_config_as_used_readonly(tmpdir): + cfg = tmpdir.join('f').ensure() + store_dir = tmpdir.join('store') + # make a store, then we'll convert its directory to be readonly + assert not Store(str(store_dir)).readonly # directory didn't exist + assert not Store(str(store_dir)).readonly # directory did exist + + def _chmod_minus_w(p): + st = os.stat(p) + os.chmod(p, st.st_mode & ~(stat.S_IWUSR | stat.S_IWOTH | stat.S_IWGRP)) + + _chmod_minus_w(store_dir) + for fname in os.listdir(store_dir): + assert not os.path.isdir(fname) + _chmod_minus_w(os.path.join(store_dir, fname)) + + store = Store(str(store_dir)) + assert store.readonly + # should be skipped due to readonly + store.mark_config_used(str(cfg)) + assert _select_all_configs(store) == [] + + +def test_clone_with_recursive_submodules(store, tmp_path): + sub = tmp_path.joinpath('sub') + sub.mkdir() + sub.joinpath('submodule').write_text('i am a submodule') + cmd_output('git', '-C', str(sub), 'init', '.') + cmd_output('git', '-C', str(sub), 'add', '.') + git.commit(str(sub)) + + repo = tmp_path.joinpath('repo') + repo.mkdir() + repo.joinpath('repository').write_text('i am a repo') + cmd_output('git', '-C', str(repo), 'init', '.') + cmd_output('git', '-C', str(repo), 'add', '.') + cmd_output('git', '-C', str(repo), 'submodule', 'add', str(sub), 'sub') + git.commit(str(repo)) + + rev = git.head_rev(str(repo)) + ret = store.clone(str(repo), rev) + + assert os.path.exists(ret) + assert os.path.exists(os.path.join(ret, str(repo), 'repository')) + assert os.path.exists(os.path.join(ret, str(sub), 'submodule')) diff --git a/tests/util_test.py b/tests/util_test.py index c9838c55..5b262113 100644 --- a/tests/util_test.py +++ b/tests/util_test.py @@ -1,43 +1,39 @@ -from __future__ import unicode_literals +from __future__ import annotations import os.path import stat +import subprocess import pytest from pre_commit.util import CalledProcessError from pre_commit.util import clean_path_on_failure from pre_commit.util import cmd_output -from pre_commit.util import parse_version +from pre_commit.util import cmd_output_b +from pre_commit.util import cmd_output_p +from pre_commit.util import make_executable from pre_commit.util import rmtree -from pre_commit.util import tmpdir def test_CalledProcessError_str(): - error = CalledProcessError( - 1, [str('git'), str('status')], 0, (str('stdout'), str('stderr')), - ) + error = CalledProcessError(1, ('exe',), b'output\n', b'errors\n') assert str(error) == ( - "Command: ['git', 'status']\n" - 'Return code: 1\n' - 'Expected return code: 0\n' - 'Output: \n' - ' stdout\n' - 'Errors: \n' - ' stderr\n' + "command: ('exe',)\n" + 'return code: 1\n' + 'stdout:\n' + ' output\n' + 'stderr:\n' + ' errors' ) def test_CalledProcessError_str_nooutput(): - error = CalledProcessError( - 1, [str('git'), str('status')], 0, (str(''), str('')), - ) + error = CalledProcessError(1, ('exe',), b'', b'') assert str(error) == ( - "Command: ['git', 'status']\n" - 'Return code: 1\n' - 'Expected return code: 0\n' - 'Output: (none)\n' - 'Errors: (none)\n' + "command: ('exe',)\n" + 'return code: 1\n' + 'stdout: (none)\n' + 'stderr: (none)' ) @@ -76,22 +72,29 @@ def test_clean_path_on_failure_cleans_for_system_exit(in_tmpdir): assert not os.path.exists('foo') -def test_tmpdir(): - with tmpdir() as tempdir: - assert os.path.exists(tempdir) - assert not os.path.exists(tempdir) - - def test_cmd_output_exe_not_found(): - ret, out, _ = cmd_output('i-dont-exist', retcode=None) + ret, out, _ = cmd_output('dne', check=False) assert ret == 1 - assert out == 'Executable `i-dont-exist` not found' + assert out == 'Executable `dne` not found' -def test_parse_version(): - assert parse_version('0.0') == parse_version('0.0') - assert parse_version('0.1') > parse_version('0.0') - assert parse_version('2.1') >= parse_version('2') +@pytest.mark.parametrize('fn', (cmd_output_b, cmd_output_p)) +def test_cmd_output_exe_not_found_bytes(fn): + ret, out, _ = fn('dne', check=False, stderr=subprocess.STDOUT) + assert ret == 1 + assert out == b'Executable `dne` not found' + + +@pytest.mark.parametrize('fn', (cmd_output_b, cmd_output_p)) +def test_cmd_output_no_shebang(tmpdir, fn): + f = tmpdir.join('f').ensure() + make_executable(f) + + # previously this raised `OSError` -- the output is platform specific + ret, out, _ = fn(str(f), check=False, stderr=subprocess.STDOUT) + assert ret == 1 + assert isinstance(out, bytes) + assert out.endswith(b'\n') def test_rmtree_read_only_directories(tmpdir): diff --git a/tests/xargs_test.py b/tests/xargs_test.py index d2d7d7b3..e8000b25 100644 --- a/tests/xargs_test.py +++ b/tests/xargs_test.py @@ -1,19 +1,52 @@ -# -*- coding: utf-8 -*- -from __future__ import absolute_import -from __future__ import unicode_literals +from __future__ import annotations import concurrent.futures +import multiprocessing +import os import sys import time +from unittest import mock -import mock import pytest -import six from pre_commit import parse_shebang from pre_commit import xargs +def test_cpu_count_sched_getaffinity_exists(): + with mock.patch.object( + os, 'sched_getaffinity', create=True, return_value=set(range(345)), + ): + assert xargs.cpu_count() == 345 + + +@pytest.fixture +def no_sched_getaffinity(): + # Simulates an OS without os.sched_getaffinity available (mac/windows) + # https://docs.python.org/3/library/os.html#interface-to-the-scheduler + with mock.patch.object( + os, + 'sched_getaffinity', + create=True, + side_effect=AttributeError, + ): + yield + + +def test_cpu_count_multiprocessing_cpu_count_implemented(no_sched_getaffinity): + with mock.patch.object(multiprocessing, 'cpu_count', return_value=123): + assert xargs.cpu_count() == 123 + + +def test_cpu_count_multiprocessing_cpu_count_not_implemented( + no_sched_getaffinity, +): + with mock.patch.object( + multiprocessing, 'cpu_count', side_effect=NotImplementedError, + ): + assert xargs.cpu_count() == 1 + + @pytest.mark.parametrize( ('env', 'expected'), ( @@ -29,19 +62,10 @@ def test_environ_size(env, expected): @pytest.fixture -def win32_py2_mock(): +def win32_mock(): with mock.patch.object(sys, 'getfilesystemencoding', return_value='utf-8'): with mock.patch.object(sys, 'platform', 'win32'): - with mock.patch.object(six, 'PY2', True): - yield - - -@pytest.fixture -def win32_py3_mock(): - with mock.patch.object(sys, 'getfilesystemencoding', return_value='utf-8'): - with mock.patch.object(sys, 'platform', 'win32'): - with mock.patch.object(six, 'PY2', False): - yield + yield @pytest.fixture @@ -81,7 +105,7 @@ def test_partition_limits(): ) -def test_partition_limit_win32_py3(win32_py3_mock): +def test_partition_limit_win32(win32_mock): cmd = ('ninechars',) # counted as half because of utf-16 encode varargs = ('πŸ˜‘' * 5,) @@ -89,13 +113,6 @@ def test_partition_limit_win32_py3(win32_py3_mock): assert ret == (cmd + varargs,) -def test_partition_limit_win32_py2(win32_py2_mock): - cmd = ('ninechars',) - varargs = ('πŸ˜‘' * 5,) # 4 bytes * 5 - ret = xargs.partition(cmd, varargs, 1, _max_length=31) - assert ret == (cmd + varargs,) - - def test_partition_limit_linux(linux_mock): cmd = ('ninechars',) varargs = ('πŸ˜‘' * 5,) @@ -143,10 +160,9 @@ def test_argument_too_long(): def test_xargs_smoke(): - ret, out, err = xargs.xargs(('echo',), ('hello', 'world')) + ret, out = xargs.xargs(('echo',), ('hello', 'world')) assert ret == 0 assert out.replace(b'\r\n', b'\n') == b'hello world\n' - assert err == b'' exit_cmd = parse_shebang.normalize_cmd(('bash', '-c', 'exit $1', '--')) @@ -154,37 +170,33 @@ exit_cmd = parse_shebang.normalize_cmd(('bash', '-c', 'exit $1', '--')) max_length = len(' '.join(exit_cmd)) + 3 -def test_xargs_negate(): - ret, _, _ = xargs.xargs( - exit_cmd, ('1',), negate=True, _max_length=max_length, - ) - assert ret == 0 - - ret, _, _ = xargs.xargs( - exit_cmd, ('1', '0'), negate=True, _max_length=max_length, - ) - assert ret == 1 - - -def test_xargs_negate_command_not_found(): - ret, _, _ = xargs.xargs(('cmd-not-found',), ('1',), negate=True) - assert ret != 0 - - def test_xargs_retcode_normal(): - ret, _, _ = xargs.xargs(exit_cmd, ('0',), _max_length=max_length) + ret, _ = xargs.xargs(exit_cmd, ('0',), _max_length=max_length) assert ret == 0 - ret, _, _ = xargs.xargs(exit_cmd, ('0', '1'), _max_length=max_length) + ret, _ = xargs.xargs(exit_cmd, ('0', '1'), _max_length=max_length) assert ret == 1 + # takes the maximum return code + ret, _ = xargs.xargs(exit_cmd, ('0', '5', '1'), _max_length=max_length) + assert ret == 5 + + +@pytest.mark.xfail(sys.platform == 'win32', reason='posix only') +def test_xargs_retcode_killed_by_signal(): + ret, _ = xargs.xargs( + parse_shebang.normalize_cmd(('bash', '-c', 'kill -9 $$', '--')), + ('foo', 'bar'), + ) + assert ret == -9 + def test_xargs_concurrency(): bash_cmd = parse_shebang.normalize_cmd(('bash', '-c')) print_pid = ('sleep 0.5 && echo $$',) start = time.time() - ret, stdout, _ = xargs.xargs( + ret, stdout = xargs.xargs( bash_cmd, print_pid * 5, target_concurrency=5, _max_length=len(' '.join(bash_cmd + print_pid)) + 1, @@ -193,16 +205,15 @@ def test_xargs_concurrency(): assert ret == 0 pids = stdout.splitlines() assert len(pids) == 5 - # It would take 0.5*5=2.5 seconds ot run all of these in serial, so if it + # It would take 0.5*5=2.5 seconds to run all of these in serial, so if it # takes less, they must have run concurrently. assert elapsed < 2.5 def test_thread_mapper_concurrency_uses_threadpoolexecutor_map(): with xargs._thread_mapper(10) as thread_map: - assert isinstance( - thread_map.__self__, concurrent.futures.ThreadPoolExecutor, - ) is True + _self = thread_map.__self__ # type: ignore + assert isinstance(_self, concurrent.futures.ThreadPoolExecutor) def test_thread_mapper_concurrency_uses_regular_map(): @@ -212,9 +223,29 @@ def test_thread_mapper_concurrency_uses_regular_map(): def test_xargs_propagate_kwargs_to_cmd(): env = {'PRE_COMMIT_TEST_VAR': 'Pre commit is awesome'} - cmd = ('bash', '-c', 'echo $PRE_COMMIT_TEST_VAR', '--') + cmd: tuple[str, ...] = ('bash', '-c', 'echo $PRE_COMMIT_TEST_VAR', '--') cmd = parse_shebang.normalize_cmd(cmd) - ret, stdout, _ = xargs.xargs(cmd, ('1',), env=env) + ret, stdout = xargs.xargs(cmd, ('1',), env=env) assert ret == 0 assert b'Pre commit is awesome' in stdout + + +@pytest.mark.xfail(sys.platform == 'win32', reason='posix only') +def test_xargs_color_true_makes_tty(): + retcode, out = xargs.xargs( + (sys.executable, '-c', 'import sys; print(sys.stdout.isatty())'), + ('1',), + color=True, + ) + assert retcode == 0 + assert out == b'True\n' + + +@pytest.mark.xfail(os.name == 'posix', reason='nt only') +@pytest.mark.parametrize('filename', ('t.bat', 't.cmd', 'T.CMD')) +def test_xargs_with_batch_files(tmpdir, filename): + f = tmpdir.join(filename) + f.write('echo it works\n') + retcode, out = xargs.xargs((str(f),), ('x',) * 8192) + assert retcode == 0, (retcode, out) diff --git a/tests/yaml_rewrite_test.py b/tests/yaml_rewrite_test.py new file mode 100644 index 00000000..d0f6841c --- /dev/null +++ b/tests/yaml_rewrite_test.py @@ -0,0 +1,47 @@ +from __future__ import annotations + +import pytest + +from pre_commit.yaml import yaml_compose +from pre_commit.yaml_rewrite import MappingKey +from pre_commit.yaml_rewrite import MappingValue +from pre_commit.yaml_rewrite import match +from pre_commit.yaml_rewrite import SequenceItem + + +def test_match_produces_scalar_values_only(): + src = '''\ +- name: foo +- name: [not, foo] # not a scalar: should be skipped! +- name: bar +''' + matcher = (SequenceItem(), MappingValue('name')) + ret = [n.value for n in match(yaml_compose(src), matcher)] + assert ret == ['foo', 'bar'] + + +@pytest.mark.parametrize('cls', (MappingKey, MappingValue)) +def test_mapping_not_a_map(cls): + m = cls('s') + assert list(m.match(yaml_compose('[foo]'))) == [] + + +def test_sequence_item_not_a_sequence(): + assert list(SequenceItem().match(yaml_compose('s: val'))) == [] + + +def test_mapping_key(): + m = MappingKey('s') + ret = [n.value for n in m.match(yaml_compose('s: val\nt: val2'))] + assert ret == ['s'] + + +def test_mapping_value(): + m = MappingValue('s') + ret = [n.value for n in m.match(yaml_compose('s: val\nt: val2'))] + assert ret == ['val'] + + +def test_sequence_item(): + ret = [n.value for n in SequenceItem().match(yaml_compose('[a, b, c]'))] + assert ret == ['a', 'b', 'c'] diff --git a/tox.ini b/tox.ini index 1fac9332..609c2fe1 100644 --- a/tox.ini +++ b/tox.ini @@ -1,14 +1,13 @@ [tox] -envlist = py27,py36,py37,pypy,pypy3,pre-commit +envlist = py,pypy3,pre-commit [testenv] deps = -rrequirements-dev.txt -passenv = HOME LOCALAPPDATA RUSTUP_HOME +passenv = * commands = coverage erase - coverage run -m pytest {posargs:tests} - coverage report --fail-under 100 - pre-commit install + coverage run -m pytest {posargs:tests} --ignore=tests/languages --durations=20 + coverage report --omit=pre_commit/languages/*,tests/languages/* [testenv:pre-commit] skip_install = true @@ -24,5 +23,6 @@ env = GIT_COMMITTER_NAME=test GIT_AUTHOR_EMAIL=test@example.com GIT_COMMITTER_EMAIL=test@example.com + GIT_ALLOW_PROTOCOL=file VIRTUALENV_NO_DOWNLOAD=1 PRE_COMMIT_NO_CONCURRENCY=1