buildbot - 如何在 Buildbot 步骤中“获取”环境？

Question

在 Buildbot 中，我需要能够在执行编译步骤之前“获取”环境。

如果我使用 bash 从命令行构建应用程序，我将不得不这样做：

. envrionment-set-up-script
build_command

在构建机器人 master.cfg 文件中，我尝试了以下操作：

factory.addStep(ShellCommand(command=["source","environment-set-up-script"])
factory.addStep(ShellCommand(command=[".","environment-set-up-script"]))
factory.addStep(Configure(command=["source","environment-set-up-script"]))
factory.addStep(Configure(command=[".","environment-set-up-script"]))

所有这些都失败了，这是因为找不到命令，这是有道理的，因为它是内置的 bash。

此外，我认为这不是正确的方法，因为在调用工厂的下一步时不一定会使用环境。

score 9 · Accepted Answer

在使用 OpenEmbedded/Yocto 时，我们以类似的方式解决了这个问题：

class EnvStep(ShellCommand):
    def __init__(self, command='', **kwargs):
        kwargs['command'] = [
            'bash', '-c', 'source environment-set-up-script; %s' % command
        ]
        ShellCommand.__init__(self, **kwargs)

然后，添加一个EnvStep将其command参数设置为foo允许我们foo在以environment-set-up-script. 换句话说，您可以通过调用来使用该步骤

factory.addStep(EnvStep(command='foo'))

并且采购将自动发生。

我们还有许多其他自定义构建步骤需要获取 build-env，因此我们只是让它们子类EnvStep化而不是ShellCommand自动处理环境。

score 5 · Accepted Answer

经过一些试验，我找到了实现这一目标的方法。你需要：

运行 bash 子 shell 设置应该用于该 shell 的环境，即调用 bash 并将环境变量BASH_ENV设置为应该被引入环境的文件。
在 bash 中运行env命令以捕获环境
将env命令的结果解析为属性（使用SetProperty步骤）
在进一步的步骤中使用该属性作为env参数

注意：环境应该被解析为可以用作环境参数的字典

    from buildbot.process.factory import BuildFactory
    from buildbot.steps.shell import ShellCommand, SetProperty
    from buildbot.process.properties import Property  

    def glob2list(rc, stdout, stderr):
        ''' Function used as the extrat_fn function for SetProperty class
            This takes the output from env command and creates a dictionary of 
            the environment, the result of which is stored in a property names
            env'''
        if not rc:
            env_list = [ l.strip() for l in stdout.split('\n') ]
            env_dict={ l.split('=',1)[0]:l.split('=',1)[1] for l in 
                          env_list if len(l.split('=',1))==2}
            return {'env':env_dict}

    #This is the equivalent of running source MyWorkdir/my-shell-script then  
    #capturing the environment afterwords.
    factory.addStep(SetProperty(command="bash -c env",
                extract_fn=glob2list,       
                workdir='MyWorkdir',
                env={BASH_ENV':'my-shell-script' }))

    #Do another step with the environment that was sourced from 
    #MyWorkdir/my-shell-script
    factory.addStep(ShellCommand(command=["env"],
                workdir="MyWorkdir",
                env=Property('env')))

score 3 · Accepted Answer

You can use the env parameter to the ShellCommand build step to setup the environment for your command. (http://docs.buildbot.net/0.8.1/Using-ShellCommands.html)

You can also set the environment before starting the build slave, if you want the environment to apply to all build command equally.

Basically either:

Work out the environment variables required and pass in in env.
Source the configuration before starting the build slave.
Wrap the source command and your builds command(s) is a shell script, and run as a single build step.

score 2 · Accepted Answer

Visual Studio 开发的示例也很有用。

通过使用通用工具脚本，为每台机器设置正确的env，同一个builder可以使用多个版本的VS，支持多种架构native和cross。

# A raw text string to get around windows quoting problems in buildbot.
vscomntools=r""""%VS120COMNTOOLS%\VsDevCmd.bat" %PROCESSOR_ARCHITECTURE% & set"""

# Determine MSVC Environment using VS Common Tools for build steps.
f.addStep(SetProperty(
    command=vscomntools,
    extract_fn=glob2list))

另一种方法是通过尝试引用批处理文件、& 和命令来启动每个命令。

score 0 · Accepted Answer

我以不同的方式解决了这个问题。我在 Windows worker 上安装了一个批处理文件，它在启动时调用环境设置批处理文件，然后执行预期的命令。当然，由于批处理文件在转发参数方面很糟糕，而且 Visual Studio 2017 的 VsDevCmd.bat 会破坏您的 cwd，因此这会变得更加困难。但是，如果你在 worker 上安装以下文件，你可以使用 VS2017 构建：

@ECHO OFF
@REM Calls a single command from the environment set from visual studio vcvars.
@REM Usage:
@REM withvcvars-2017.bat <arch> [command]
@REM 
@REM Run VsDevCmd.bat /help to see available arches
@REM See below instantiation for more fine grained option setting.

set ARCH=%1
shift

setlocal enabledelayedexpansion

@rem Replace __ with = in batch files.
@rem This works around idiotic lack of equals signs in args
@rem args contains full args string with substitutions in place
:argloop
if "%~1" NEQ "" (

    set str=%~1
    set out=!str:__==!
    set %~1=!out!
    set args=!args!!out! 

    SHIFT
    goto :argloop
)

@rem Aside from batch files being too moronic to allow for equals signs in args,
@rem VsDevCmd.bat idiotically clobbers the current working directory when it's called.
set CWD=%cd%

echo Calling VsDevCmd.bat for arch %ARCH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=%ARCH% -winsdk=8.1 -app_platform=Desktop -no_logo

@rem Who lets these things out the door?
cd %CWD%

@ECHO ON
%args%

完成此操作后，您可以在 bulidbot 主逻辑中创建一个附加此批处理文件的函数：

def _withvcvars(year, arch, cmd):     
    base_cmd = ["%swithvcvars-%s.bat" % ('\\path\\to\\batchfile\\', year), arch]      
    return base+cmd

这使您可以运行调用 msbuild.exe 的命令，该命令在其参数中需要等号。只需将它们指定为双下划线：

withvcvars-2017.bat amd64 msbuild.exe your.sln /p:Configuration__Release /p:Platform__x64

score 0 · Accepted Answer

    steps.ShellCommand(
            name = "Example",
            command = ["/bin/bash", "-c", "source <BASH SCRIPT>"],
            haltOnFailure = True,
            description = "Example"
    ),

buildbot - 如何在 Buildbot 步骤中“获取”环境？

6 回答 6

Related

Reference