1. дома
  2. Вопрос и ответ
  3. компилирования Hadoop Native ЛИЭС из источника на Mac OS

компилирования Hadoop Native ЛИЭС из источника на Mac OS

2016-11-26 2 views
2

Я strugling с составлением последней Hadoop от источника со следующим исключением:компилирования Hadoop Native ЛИЭС из источника на Mac OS

[INFO] Executed tasks 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-pipes --- 
[INFO] 
[INFO] --- hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-pipes --- 
[INFO] Running cmake /Users/romeo/Repository/hadoop/hadoop-tools/hadoop-pipes/src -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles 
[INFO] with extra environment variables {} 
[WARNING] CMake Error at /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:138 (message): 
[WARNING] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the 
[WARNING] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR) 
[WARNING] Call Stack (most recent call first): 
[WARNING] /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE) 
[WARNING] /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindOpenSSL.cmake:385 (find_package_handle_standard_args) 
[WARNING] CMakeLists.txt:24 (find_package) 
[WARNING] 
[WARNING] 
[WARNING] -- Configuring incomplete, errors occurred! 
[WARNING] See also "/Users/romeo/Repository/hadoop/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log". 
[INFO] ------------------------------------------------------------------------ 
[INFO] Reactor Summary: 
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.920 s] 
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.939 s] 
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.191 s] 
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.458 s] 
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.170 s] 
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.170 s] 
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 6.001 s] 
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 2.502 s] 
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 7.454 s] 
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.214 s] 
[INFO] Apache Hadoop Common ............................... SUCCESS [ 56.405 s] 
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 5.317 s] 
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 7.725 s] 
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.044 s] 
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 15.292 s] 
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 39.539 s] 
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 2.075 s] 
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 22.573 s] 
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 3.311 s] 
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.049 s] 
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.044 s] 
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.825 s] 
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 22.424 s] 
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.030 s] 
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 5.927 s] 
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 8.990 s] 
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.795 s] 
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 4.752 s] 
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 7.784 s] 
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.938 s] 
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.469 s] 
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 4.031 s] 
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.766 s] 
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 2.808 s] 
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 2.918 s] 
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.041 s] 
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.676 s] 
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 2.113 s] 
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.043 s] 
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.960 s] 
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.029 s] 
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 6.991 s] 
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.138 s] 
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 14.384 s] 
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 12.110 s] 
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 3.040 s] 
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 6.127 s] 
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.264 s] 
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 3.597 s] 
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.715 s] 
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 6.028 s] 
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.248 s] 
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.660 s] 
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 3.836 s] 
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 3.238 s] 
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.621 s] 
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 2.315 s] 
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.120 s] 
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.797 s] 
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.888 s] 
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.628 s] 
[INFO] Apache Hadoop Pipes ................................ FAILURE [ 0.533 s] 
[INFO] Apache Hadoop OpenStack support .................... SKIPPED 
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED 
[INFO] Apache Hadoop Azure support ........................ SKIPPED 
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED 
[INFO] Apache Hadoop Client ............................... SKIPPED 
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED 
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED 
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED 
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED 
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED 
[INFO] Apache Hadoop Tools ................................ SKIPPED 
[INFO] Apache Hadoop Distribution ......................... SKIPPED 
[INFO] ------------------------------------------------------------------------ 
[INFO] BUILD FAILURE 
[INFO] ------------------------------------------------------------------------ 
[INFO] Total time: 06:12 min 
[INFO] Finished at: 2016-11-26T07:18:34+01:00 
[INFO] Final Memory: 146M/857M 
[INFO] ------------------------------------------------------------------------ 
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 -> [Help 1] 
[ERROR]

Как видно из вывода он терпит неудачу на «Apache Hadoop Pipes», потому что неразрешенной зависимости «Плагины Apache Hadoop Maven». Тем не менее, плагины Maven были выполнены с состоянием SUCCESS:

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT -> [Help 1] 
org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT 
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:128) 
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor(DefaultMavenPluginManager.java:179) 
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor(DefaultMavenPluginManager.java:284) 
    at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor(DefaultBuildPluginManager.java:241) 
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.setupMojoExecution(DefaultLifecycleExecutionPlanCalculator.java:169) 
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.setupMojoExecutions(DefaultLifecycleExecutionPlanCalculator.java:155) 
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:131) 
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:145) 
    at org.apache.maven.lifecycle.internal.builder.BuilderCommon.resolveBuildPlan(BuilderCommon.java:96) 
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:109) 
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) 
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) 
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) 
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) 
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) 
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) 
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) 
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) 
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:497) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) 
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) 
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444) 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246) 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223) 
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:294) 
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:124) 
    ... 26 more 
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT 
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:434) 
    ... 30 more 
[ERROR]

Любые советы или подсказки были бы оценены. Благодарю.

Update1: Установка OpenSSL сделал решить главную проблему:

[INFO] ------------------------------------------------------------------------ 
[INFO] Building Apache Hadoop Pipes 3.0.0-alpha2-SNAPSHOT 
[INFO] ------------------------------------------------------------------------ 
[WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT is missing, no dependency information available

UPDATE2: После выполнения mvn install получать новое исключение:

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 -> [Help 1] 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) 
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) 
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

источник

2016-11-26 Roman Melnyk

ответ

1

Установка hadoop-maven-plugin может быть сделано mvn install первым.

$ mvn clean install -DskipTests 
$ mvn package -Pdist -Pnative -Dtar -DskipTests

Детали зависимостей описаны в BUILDING.txt.

Или, если у вас есть докерная среда, start-build-env.sh создает контейнер докеров, на котором можно построить Hadoop.

Поскольку все необходимые зависимости уже установлены в этом контейнере, вы можете создать пакет Hadoop, включая встроенный внутри этого контейнера.

источник

2016-11-26 08:32:24

+0

спасибо. Но мне это очень не помогло. :) [ERROR] Не удалось выполнить цель org.apache.hadoop: hadoop-maven-plugins: 3.0.0-alpha2-SNAPSHOT: cmake-компиляция (cmake-компиляция) в проекте hadoop-pipe: ошибка CMake с кодом ошибки 1 -> [Помощь 1] –

+0

Вы установили CMake в свой MacOSX? Или как насчет использования 'start-build-env.sh'? –

0

Ответ here

Проблема была связана с пакетом OpenSSL. По умолчанию он не включает заголовки dev.

источник

2016-11-26 13:30:53

Смежные вопросы

 Смежные вопросы